NetTiers Ambiguous match found

I recently ran into a bug that took me longer than it should have to figure out.

I was migrating an old code base to CodeSmith's NetTiers and started getting the error "Ambiguous match found".  Results from the Internet all pointed the finger at Microsoft's .net, albeit earlier versions than version 4.5 that I was using.

While I did become convinced that it was a Microsoft issue relating to two object attributes with the same name just different letter casing, that did not solve my issue for me because my code was all being auto generated by NetTiers.  NetTiers was using reflection to get some attribute values, so I figured one of my table or column names was conflicting with some internal variable that NetTiers was using.

The solution turned out to be rather simple.  I ran the code in debug mode, triggered the crash, and started inspecting the variables in play.  Particularly the one NetTiers was trying to find using reflection.  It turned out that NetTiers was looking for an attribute named "Item", which corresponded to a table column of the same name, and was blowing up in the attempt.  I renamed that column in my database, regenerated the code, and the problem was solved.

Downloading Ancestry.com Media

Ancestry.com has never created a good system to back up the data that is on their site. The only option people have is to download the GEDCOM file, which is often a mere fraction of the data they have invested in the site.

For years now my family has been investing time in building a well documented family tree on ancestry.com, and I have made sure to frequently backup the GEDCOM file so we didn't lose the data.  With thousands of media files on the site I have become increasingly concerned about the loss of a major portion of our research if I was unable to download them.

To combat this problem I finally wrote a small Chrome extension that allows you to screen scrape your media files in an automated fashion.  It is not perfect, but considering it is the only way to get your data, it is nice to have.

The extension downloads the files in mass and dumps them all into your default downloads folder.  So, you should clear all files out of your default downloads folder before using this extension so you know that all downloaded files belong to Ancestry.

After installing this extension, you need to:

- log into your ancestry.com account

- open up your tree

- open up the Media page, which is one of the items on the menu inside the tree

- click on either the Photos or Stories tab, currently the All, Audio, and Video tabs are not supported

- OPTIONAL: click on the page you want to resume downloading at.  This is useful if you need to stop downloading for a period of time, say if you only want to run the downloads during the night.  You will likely end up with a few duplicate files if you attempt this.

- click on the Extension icon in your Chrome browser upper right corner, and click the Download Media button

If your browser is asking you to save every file then the extension will not be able to correctly calculated the required download time and it will go on to the next page before you are done. It is expecting the download to automatically start.  To fix this problem, go into your browser settings and adjust it so downloads do NOT always prompt you for a save location. Instead set a default location for it to use every time. The only way I could fix this issue would be to create a setting that allows the user to manually click the next page button instead of having it automated. And I think that would kind of defeat the point of the extension.

If the downloads are occurring too quickly and swamping your computer then let me know.  I am planning on allowing the download speed to be adjusted in the future.  Currently a file is downloaded every 2 seconds, which works well for me, but for people who upload lots of large files it will not work as well for.

NOTE: There is a small chance that using this is against Ancestry's terms of use.  Their terms of use forbid scraping tools, however the language seems directed at automated scripts that run in the background.  This tool is simply a browsing aid performing clicks for you, but your browser is still open and doing all the browsing and preventing you from doing other browsing simultaneously.

EDIT 10/2018:

Just released a new version that works with Ancestry's latest photo hosting method.  In order to get it working again the extension is going to need permissions to open and close windows as well as download files.  When you first try and use the photo downloading feature Chrome will likely popup a few requests asking you to let the extension do downloads and open and close windows.  So expect that as you try to get it going.

This version also handles the Start/Stop downloading sensing better.  Although, I have noticed that sometimes it takes one or two tries to get it to stop downloading depending on at what point in its execution you click the stop button.

LastPass Password Manager

Most people I have met know that long complex passwords are a good idea.  But very few people know of a good way to keep track of all their passwords.  I personally have well over 200 accounts that I need to track passwords for.  Many people, myself included, get stuck in the habit of picking out a handful of passwords and using them for everything.

Over the years I have tried a few password managers including KeyPass and PassPack.  They all had nice features, but they all ended up being such a pain that I was never able to use them to create truly random unique passwords for every account I have.  KeyPass did not sync across computers, PassPack had a very limited free version, and the issues go on.

Then I discovered LastPass.  It took me a month of usage to fully start trusting it, but slowly I have started using more and more features of it.  The best part is the amount of features they offer for free.  And their first paid level ( that many people will want ) is only $12/yr at the moment.  So price for features was exactly what I was looking for.

LastPass syncs across all my computers, or phones for free (paid version can do both at the same time).  It auto fills my website passwords with a Chrome plugin.  It has been gamified to show me my overall security score and compare it to the score of other users.  It supports lots of dual factor authentication options.  It allows me to re-prompt for my password and do other security enhancements on a per password basis.  It even has built in form fills, both generic and tailored to sites that require more than just a username and password to login.  And critically, it allows me to backup all the data in my account.

Having an easy backup system is critical for me, because no matter how good software is, and how much they claim they are not going anywhere, anything can happen.  Their website could crash, they could go out of business, my encrypted data could get corrupted, I could forget my master password, etc.  So I use the backup feature frequently to export a plain text copy of all my data including passwords, I then encrypt the file and store it.  So if anything every happens, or a better software comes along, I can easily recover using my backup file.

In addition to a good backup system, I also wanted to be sure they could not be hacked.  Unfortunately they have been hacked at least twice.  However, the articles and responses to these hacks were so good that it made me feel even more secure using the software; no critical data was lost, users could easily remove the danger by changing the one piece of lost data, and that lost data was encrypted so heavily that it was unlikely anyone would figure it out anyway.  I did use their advanced settings to increase my security above the default level though.

A few things I did to make my account more secure:
- Make sure all my client browser plugins log me off after a reasonable period of inactivity
- Go into my vaults advanced settings:
- - increase the client-side rounds to something greater than 50k. Due to LastPass's speed warnings I did this in increments testing the speed both on my phone and computer.  I also backed up my data first in case the re-encryption corrupted everything.
- - restricted logins to just my country
- - added dual factor authentication
- - reduced my website logoff settings down to a day

Studying for Microsoft 70-463 ( Implementing a Data Warehouse with Microsoft SQL Server 2012 )

For those of you out there who wish to get your MCSA SQL Server 2012, this is my path of choice, I hope it helps some of you out:

1. Take the free Jump Start course from MVA
2. Practice and brush up on features using the Microsoft Virtual Lab for Exploring MDS and DQS, about half way down the page.
3. Pull together 60 practice questions. which you can take as well.
4. Pull together 60 more practice questions.
5. Pull together 60 more practice questions.
6. Pull together a few more practice questions.
7. Found someone else's 86 practice questions.
8. Buy the MeasureUp practice exam and hammer it to death.
9. Go take the test.

Microsoft even offers their own practice test for this exam if you wish to purchase it.  The real exam may or may not include any of the questions compiled in any set of practice questions.

That said, practice tests are only a means of validating a persons knowledge.  The MVA class and Virtual Lab are where the actual learning should take place.

Cloning / Hacking RF remote controls using Arduino

I've been on a smart house kick lately trying to add every sensor I can find to my house that falls within my budget.  Due to budgeting issues I am always looking for new ways to do things cheaper.

Basic sirens coupled with Normal Open and Normal Closed sensors were the obvious first step because they are cheap and fairly easy to understand and implement.  And they give you rudimentary detection and feedback capabilities.  But I want more, and to get more I have been exploring all the IP based options available for controlling lights, outlets, and other household devices.

Unfortunately, in part to the current smart house craze, any device that supports traditional smart house technology like z-wave, wifi, and zigbee are extremely expensive.  But then I thought about traditional remote controls.

The vast majority of remotely controlled devices use standard Radio Frequencies (RF), usually in the 315 or 433MHz range.  Most people do not consider this to be smart house related because they only work up to a range of a few hundred feet, are usually unidirectional in communication, have no interface to be controlled through the Internet, and absolutely zero security around the protocol.

There isn't much I can do about the lack of security, that's just a fact of life about cheap remote controls.  However, I did not see any reason why I could not make a computer connected radio that sent out the same wave patterns as my remotes.  With these cloned signals I would then be able to remotely and centrally control anything with an RF receiver in it.

After days of reading about the subject and other peoples attempts to do the same thing, I bought an arduino and a 315MHz receiver/transmitter set.  Three days later after a lot of signal analysis, trial and error, I finally got my first signal cloned.  I now have the ability to repeat the process for any RF remote that operates on the 315MHz frequency with a fixed code and control its item remotely as a fully integral part of my smart house.

The process involved:
- writing a program for arduino that sampled one of the analog pins on a tight loop tracking the strength and duration of the signal fluxuations.
- I then moved this data into excel where I normalized it to determine the binary code it was sending.
- Converted that binary code into an integer mapping table for condensed storage.
- Wrote a program to use a digital out pin to send that signal to the transmitter which turned it back into an analog wave form in the air.

To be fair, both of the base programs I started with someone else who knew the arduino far better wrote, I just modified them to be more efficient for my use case.

Here is the code for the RF Receiver to capture signals:

 #define analogPin A0 // RF data pin = Analog pin 0  
 byte dataBuffer[512];  
 int dataCounter = 0;  
 int maxSignalLength = 255;  
 const unsigned int upperThreshold = 100;  
 const unsigned int lowerThreshold = 80;  
 void setup() {  
  Serial.begin(115200);  
  // wait until a LOW signal is received  
  while(analogRead(analogPin) < 1) { }  
  // got HIGH; read the rest of the data into dataBuffer  
  for (int i = 0; i < sizeof(dataBuffer); i = i+2) {  
   // LOW signals  
   dataCounter = 0;  
   while (analogRead(analogPin) > upperThreshold && dataCounter < maxSignalLength)  
    dataCounter++;  
   dataBuffer[i] = dataCounter;  
   // HIGH signal  
   dataCounter = 0;  
   while(analogRead(analogPin) < lowerThreshold && dataCounter < maxSignalLength)  
    dataCounter++;  
   dataBuffer[i+1] = dataCounter;  
  }  
  Serial.println("LOW,HIGH");  
  delay(20);  
  for (int i = 0; i < sizeof(dataBuffer); i = i+2) {  
   Serial.print(dataBuffer[i]);  
   Serial.print(",");  
   Serial.println(dataBuffer[i+1]);  
   delay(20);  
  }  
 }  
 void loop() { }  

Receiving the signals is pretty straight forward.  You have to wire your receiver into the arduino.  If you are holding it with the top of the board facing you, then the pins are from left to right: Power, Data, Data, Ground.  Hook the Power (or Vcc) into a 5V pin, 3.3V should work as well.  This code is supposed to be used with the second Data pin, but it doesn't really matter, the HIGH/LOW values will just be reversed if you use the other pin.

Upload the code to the arduino, it will automatically run since it is all in the setup function.  Press Ctrl+Shift+M to bring up the serial monitor, and make sure you set it to receive at 115200.

Immediately press the button on your remote.  Once the data appearing on the screen it is too late, the capture has already taken place.  I had to do this process several times, playing with the data each time until I finally got a capture that was clean enough I was able to find a good pattern in it.

Next you need to paste the data into excel, or some other text editor and start manipulating it.  You need to split it into two columns, a HIGH and  LOW column, then count the number of occurrences of every value one column at a time.  I wrote a macro to do this for me (the guy who's tutorial I followed wrote a php script for himself).

Next sort the value low to high based on the value that was being counted.

This is the really tricky part now.  You need to identify the signal groupings.  Most basic remotes are only going to have two.  A common short burst and a common long burst; or smaller number and larger number.  So pick the low value and the high value that occur most frequently and group the other more random values in on those two.  I picked the values manually, but had Excel re-write the originally values into two new "normalized" columns for me.

Then you re-combine the two new "normalized" columns back into a single column, perform a Unique function on them, and number each unique value.  The purpose for this is we need an easy way to identify the values, copying and pasting all those values pairs into code over and over would just get too difficult very fast.

Finally you make another column which pairs the two "normalized" columns up with their newly assigned unique value.  Combine all those values into a comma delimited string.  And start searching for patterns.  I often find that the unique value column is a better place for me to go pattern hunting in.  Values of 0 and 255 are timeouts and can usually be ignored; if they happen in a very specific pattern inside of the pattern you end up identifying then it could indicate a missing value that you need or could try guessing at.

Here is what my excel sheet looked like through one of these processes:

And here is the Code to re-transmit the signal, however it will have to be modified for your values:

 #define transmitPin 7 // RF transmit pin = Digital pin 7  
 int ding[] = {1,2,3,2,3,2,3,2,2,2,2,2,3,2,3,2,3,2,2,3,3,2,2,2,2};  
 // Play with this value  
 int timeDelay = 155;  
 int signalPin = 13;  
 int numberOfTransmissions = 19;  
 int numberOfSignalInTransmission = (sizeof(ding)/sizeof(int)) - 1;  
 void setup() {  
   Serial.begin(9600);  
   pinMode(transmitPin, OUTPUT);  
   pinMode(signalPin, OUTPUT);  
   delay(2000);  
  // wait 3 seconds, then transmit the payload  
  int highLength = 0;  
  int lowLength = 0;  
  for(int y = 105; y < 415; y=y+10) {  
  for(int x = 0; x < numberOfTransmissions; x++) {  
   if(x % 2 == 0)  
    digitalWrite(signalPin, HIGH);   
   else  
    digitalWrite(signalPin, LOW);  
   for(int i = 0; i < numberOfSignalInTransmission; i++) {  
    switch(ding[i]) {  
     case 1:  
      lowLength = 3;  
      highLength = 119;  
     break;  
     case 3:  
      lowLength = 3;  
      highLength = 11;  
     break;  
     case 2:  
      lowLength = 11;  
      highLength = 3;  
     break;  
     case 4:  
      lowLength = 5;  
      highLength = 5;  
     break;  
     case 5:  
      lowLength = 42;  
      highLength = 2;  
     break;  
    }  
    // transmit HIGH signal  
    digitalWrite(transmitPin, HIGH);    
    delayMicroseconds(highLength*y);  
    // transmit LOW signal  
    digitalWrite(transmitPin,LOW);    
    delayMicroseconds(lowLength*y);   
   }  
  }  
  }  
 }  
 void loop() { }  

The transmitter comes with 3 or 4 pins depending on if it has an onboard antenna or not.  The antenna is absolutely critical; I had a receiver that came without one and it would rarely pick up any signals unless I held a bit of mettle over the solder point where the antenna was supposed to be.

The pins from left to right if you are holding it facing you are: Digital receiver (hook to digital pin 7 for the code above), Power (Vcc 3.3V or 5V, they can usually go up to 12V for increased range), Ground.

The code above is designed with three nested for loops.

The inner most simply takes the ding[] array, converts it back into the values from the normalized columns in your excel document using a case statement, and transmits them. I did not need the last two case statements in this instance.

The next one out specifies how many times you want to send the pattern; most remotes transmit a repeating pattern several times so there is no point is storing multiple copies of the same pattern if we can just loop through it several times.

The outermost loop is very important.  It adjusts the length of the signals you are sending out, how long each high/low signal should be held for before moving on to the next.  The remotes I have seen use a value in the high 200s or low 300s for this.  Finding the precise number can be quite difficult though, so looping through a range that increments by 10 microseconds is a much faster way to at least figure out if you are on the right track before trying to narrow the signal down.

------------------------
Unfortunately it looks like the website I used to figure this all out has gone offline, which is a bummer because it had very detailed step-by-step instructions and was the only set of instructions I could find that I was able to understand clearly enough to figure this all out.

I believe this is the original site I got my tutorial from that is no longer active.
However, here is another site that looks almost as good.

3D printer - Prusa i3 - Repetier - Basic Configuration

When I first built my Prusa i3 I got lucky enough to find someone who had uploaded a pre-configured copy of the repetier firmware.  I uploaded his firmware to my printer and it just worked, like magic.

Unfortunately I ran into a hickup when helping my son with his arduino based RC car.  I accidentally uploaded his arduino program to my printer.  And when I looked for the pre-configured firmware again I was unable to find it.

So, I downloaded the latest and greatest repetier firmware with all default parameters, uploaded it, and kept my fingers crossed.  After about a week of experimentation I realized that my end stops were the only thing not functioning correctly, they seemed to be placed on the wrong side of the axis for some reason.

Another week of playing with settings and no success I finally ran across this post.  All this time I had been focused on my Endstop configuration, it had never occurred to me that was not where the problem lay.

I jumped down to the XYZ movements area, located :
#define INVERT_X_DIR 0
#define INVERT_Y_DIR 0
#define INVERT_Z_DIR 0

and changed them to:
#define INVERT_X_DIR 1
#define INVERT_Y_DIR 1
#define INVERT_Z_DIR 1

And I was back in business, everything was working they way it was before.  This time I backed up my configuration in case this happens again.