Installing VeraCrypt on Raspberry Pi4

Raspberry Pi 4 Project

I have been interested in small computing devices for awhile starting with various Arduino projects.  Recently I took on a new Raspberry project where I needed it to be able to read encrypted USB drives.  I ran into several issues along the way that I intend to document here to help others attempting similar things.

When working with a PI I generally use the Raspbian OS, it has been the one I have had the most success with.  For this project I tried both the Lite and the Full versions of the OS, and while they were both successful, the full version had one feature that caused me to go with it over Lite.

USB exFat Issue

The USB drives I was using were 64GB, which means they were too big for the older FAT32 format and were formatted exFat.  The full version of Raspbian auto mounted and read them perfectly, while the Lite version could not and needed the exFAT libraries installed using the following two commands:

sudo apt-get install exfat-fuse
sudo apt-get install exfat-utils

I didn't have any problem installing the libraries, but after install the drives would still not auto mount as the documentation indicated they should.  They did work correctly after performing a manual mount  using the following command:

sudo mount -t exfat /dev/sda1 /mnt/usb1

Part of this project was having an environment that wouldn't take me too long or too much work to re-create in the future if my micro SD card died.  So rather than continue to fight with why the USBs were not auto mounting, I ditched my preferred Lite version of Raspbian, and went with the Full version.

VeraCrypt

Now that my external USB drives were working correctly "out of the box", all I really needed to do was install the VeraCrypt software.  I use VeraCrypt because it is both free and one of the simplest and most portable encryption options available.

The first place I looked for a package was the recent downloads.  I grabbed the armhf version which is advertised as working on the 32bit Raspbian OS.  After installing and attempting to run it I was presented with the following errors:

veracrypt: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by veracrypt) veracrypt: /usr/lib/arm-linux-gnueabihf/libwx_gtk3u_core-3.0.so.0: version `WXU_3.0.5' not found (required by veracrypt)

I founded one post that indicated the errors meant I had the wrong version of the libraries installed.  It took me awhile to figure out the actual names of the libraries the dependencies were referring to, however Code Yarns finally solved that problem for me telling me to run this command.

sudo apt install libfuse-dev libwxbase3.0-dev

Unfortunately I was still getting the same error.  Considering Ashwin from Code Yards was successful in getting it to work, and remembering the errors were supposed to be version issues, not just if the libraries were installed at all, I decided to see if the version of VeraCrypt Ashwin used would work.

But of course I didn't want to go all the way back to his version, so I started walking my way back trying each version to see if one would work.  Not a particularly fun task as VeraCrypt is not compiled for Raspian with each iteration, you have to go into each folder and see if you can find an armhf version compiled.  Several people solved this issue by simply compiling the latest version of VeraCrypt for themselves; but again, I was trying to go for the most easily reproducible build.

I finally found a working version in VeraCrypt 1.24 Update 7.  My guess is that if I had looked for newer versions of libfuse and libwxbase I might have been able to use a newer version of VeraCrypt as well.

After that it was all down to testing the install and making sure encrypted files could be created and opened.  I had no more issues afterwards, the Raspberry Pi device is really becoming quite the powerful tool these days.

SQL Server fast failover in High Availability Groups

 I recently had the somewhat dubious honor of helping migrate a company from a physical data center to the Azure cloud.  I say dubious because for as many positives as there are about Azure, there are at least the same number of negatives.  The only feature Azure has that makes it worth migrating, in my opinion, for most smaller companies is the ability to work and see the environment the same way a data center would see it; so much more visibility, not good visibility, just visibility.

However, my feelings about Azure are not the point of this post.  As a part of this migration we went from a traditional failover SQL cluster to an Always on High Availability group; primarily because the dba doing the migration could not figure out how to do a failover cluster in Azure.

In my short exposure to the high availability option it is both good and bad.  The automated failover in SQL server is faster than it was using the old method, however the visibility into the individual server databases is worse.

The DBA handling the migration was unable to figure out how to manually fail back the system in a reasonable time frame.  He was using a SQL script that failed back one availability group at a time.  This approach would leave our system down for up to a minute, a time frame that would increase as we added more databases to the system, and this was a big problem.

To solve this problem I decided to go with a multi-threaded approach so I could fail back every availability group simultaneously.  Unfortunately SQL does not seem to have any form of reasonable multi-threaded capability available for a script to use.  So I turned to scripting languages.

While this script could be written in most scripting languages, I chose the older VBscript this time.  The general idea is that the script loops through all found availability groups, and spins up an instance of itself in a separate thread to start a failover for each one.

Const sqlUser = "<username>" ' add your server admin login username here

Const sqlPass = "<password>" ' add the password for the username here

Const Server = "<ip address>" ' enter the ip address of the DESTINATION server to fail to here

Const adOpenStatic = 3

Const adLockOptimistic = 3

'

' Here is where we fail over the current AG

'

If (WScript.Arguments.Count > 0) Then

    'Wscript.Echo WScript.Arguments.Item(0)

Set failConnection = CreateObject("ADODB.Connection")

    failConnection.Open _

        "Provider=SQLOLEDB;Data Source=" + Server + ";" & _

            "Trusted_Connection=No;Initial Catalog=master;" & _

                 "User ID=" + sqlUser + ";Password=" + sqlPass + ";"

failConnection.Execute "ALTER AVAILABILITY GROUP " + WScript.Arguments.Item(0) + " FAILOVER;"

failConnection.Close

Set failConnection = Nothing

    WScript.Quit 0

End If

'

' Below is where we get the list of AGs to fail over

'

Dim FSO

Set FSO = CreateObject("Scripting.FileSystemObject")

GetCurrentFolder = FSO.GetAbsolutePathName(".")

Set objConnection = CreateObject("ADODB.Connection")

Set WshShell = WScript.CreateObject("WScript.Shell")

objConnection.Open _

    "Provider=SQLOLEDB;Data Source=" + Server + ";" & _

        "Trusted_Connection=No;Initial Catalog=master;" & _

             "User ID=" + sqlUser + ";Password=" + sqlPass + ";"

Set objRecordSet = objConnection.Execute("SELECT name FROM master.sys.availability_groups")

objRecordSet.MoveFirst

Do Until objRecordSet.EOF

    'Wscript.Echo objRecordSet.Fields("name")

WshShell.Run Wscript.ScriptFullName + " " + objRecordSet.Fields("name")

objRecordSet.MoveNext

Loop

objRecordSet.Close

objConnection.Close

Set objConnection = Nothing

Wscript.Echo "All failover commands fired"

With this approach the entire fail back is reduced to a couple of seconds, and the time does not increase noticeably as additional groups get added in the future.

Converting Google Spreadsheet App Script to Libre Office Macro

 I recently had a task to convert a small macro that did some data copying in a Google Spreadsheet, into an equivalent version in Libre Office.

Google has a very JavaScript like feel to it's scripting language, while Libre/Open Office uses either Python or Basic as their languages.  Basic seemed like a closer format to what I was coming from, and I've had some experience with VBA in the past, so I chose to go with that over the more powerful Python alternative.

My primary purpose in posting this is that there seems to be a lack of documentation around the Libre Office Basic language, which is understandable given the support of Python.  But in case someone else really feels like using Basic for a small project like I did, hopefully this will help them find some of the functions they need.

Here is the Google macro I was trying to convert

function fCopyData() {
  // get all the required sheets
  var shtSum = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Source");
  var shtFut = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Dest");

  // get the values we will be working with
  var vals = shtSum.getRange("A7:D45");
  var vIn = shtSum.getRange("I2:K10");

  // if we don't have enough rows, add more
  if((shtTran.getMaxRows()-shtTran.getLastRow()) < vals.getNumRows()) {
    shtTran.insertRows(shtTran.getMaxRows(),vals.getNumRows()-(shtTran.getMaxRows()-shtTran.getLastRow())+2);
  }

  if((shtFut.getMaxRows()-shtFut.getLastRow()) < vals.getNumRows()) {
    shtFut.insertRows(shtFut.getMaxRows(),vals.getNumRows()-(shtFut.getMaxRows()-shtFut.getLastRow())+2);
  }

  // populate dest data
  var today = new Date();
  today.setDate(today.getDate()+7);

  var x = 1;
  while(x <= vals.getNumRows()) {
    if(vals.getCell(x,1).getValue()=="ENV") {
      var lastRow = shtFut.getLastRow()+1;
      shtFut.getRange(lastRow,1).setValue(vals.getCell(x,4).getValue()*-1); // col 1
      shtFut.getRange(lastRow,2).setValue((today.getMonth()+1)+'/1/'+today.getYear()); // date
      shtFut.getRange(lastRow,3).setValue('=Year(B'+lastRow+')'); // year
      shtFut.getRange(lastRow,4).setValue('=Month(B'+lastRow+')'); // month
      shtFut.getRange(lastRow,5).setValue(vals.getCell(x,2).getValue()); // category
      shtFut.getRange(lastRow,6).setValue(vals.getCell(x,3).getValue()); // description
    }
    x++;
  }

  Browser.msgBox("Finished!");
}

Google has that special feature where you actually have to add new rows when you want to use them in their spreadsheets.  Moving to a desktop app no longer had that restriction, so part of the code could just be removed which was nice.

However, figuring out all the correction function names to take the place of the Google equivalents proved to be very difficult.

So here is the equivalent in Libre/Open Office Basic

REM  *****  BASIC  *****

Sub CopyData
  Dim Document As Object
  Dim Sheets As Object
  Document = ThisComponent  'assigns the current document to the variable document
  Sheets = Document.Sheets  'get the container of all Sheets

   ' get all the required sheets
  Dim shtTran As Object
  Dim shtSum As Object
  Dim shtFut As Object
  shtSum = Sheets.getByName("Source")
  shtFut = Sheets.getByName("Dest")

  ' get the values we will be working with
  Dim vals As Object
  Dim vIn As Object
  vals = shtSum.getCellRangebyName("A7:D45")
  vIn = shtSum.getCellRangebyName("I2:K10")

  ' populate dest data
  Dim thisYear As String
  Dim thisMonth As String
  Dim lastRow As Integer
  thisYear = Format(Now() + 7, "YYYY")
  thisMonth = Format(Now() + 7, "MM")

  oCursor= shtFut.createCursor
  oCursor.gotoEndOfUsedArea(False)
  lastRow = oCursor.RangeAddress.EndRow

  Dim Cell as Object
  Dim x As Integer
  For x = 0 To (vals.Rows.getCount() - 1)
    If vals.getCellByPosition(0,x).String = "ENV" Then
      lastRow = lastRow+1

      ' col 1
      Cell = shtFut.GetCellByPosition(0,lastRow)
      Cell.Value = vals.getCellByPosition(3,x).VALUE*-1
      Cell.NumberFormat = 125 ' secret code for Currency Format

      ' date
      Cell = shtFut.GetCellByPosition(1,lastRow)
      Cell.Value = DateValue(thisMonth & "/1/" & thisYear)
      Cell.NumberFormat = 36 ' secret code for Currency Format

      ' year
      Cell = shtFut.GetCellByPosition(2,lastRow)
      Cell.Formula = "=YEAR(B"+(lastRow+1)+")"
      Cell.NumberFormat = 0 ' Numeric Format

      ' month
      Cell = shtFut.GetCellByPosition(3,lastRow)
      Cell.Formula = "=Month(B"+(lastRow+1)+")"
      Cell.NumberFormat = 0 ' Numeric Format

      ' category
      Cell = shtFut.GetCellByPosition(4,lastRow)
      Cell.String = vals.getCellByPosition(1,x).String

      ' description
      Cell = shtFut.GetCellByPosition(5,lastRow)
      Cell.String = vals.getCellByPosition(2,x).String
    End If
  Next x

  msgbox "Finished!"
End Sub

Milestone Screen Recorder changes Default username

 I've been using the Milestone XProtect VMS system for several years now, and it has had various issues, many of which they have solved over time.

One of their addons, called the Screen Recorder, is a particularly handy piece of software, but has caused an outsized number of headaches given the simplicity of it.

It allows you to record a computer screen pretending as though the computer screen were a camera so all the recording footage is stored along side of all your other video footage in your VMS system.

There are multiple very helpful uses for it, one of the best is in recording customer facing demo systems to make sure people aren't doing things they should not be.

For many years it had a very odd bug in it where if the computer was turned off for too long (days), then the recording server would eventually stop trying to contact it, so when the computer was turned back on, the recording process would not be re-initiated.  However, a recent fix seems to have solved that bug.

A month ago I upgraded my demo system to the 2020R2 version of the recording server, and for the last couple of weeks I have been struggling trying to get a couple of screen recorders to come back online.  Traditionally in the past the big "gotcha" with the screen recorder has been that the username is hard coded to "Milestone" with a capital M.  If you don't know that then you will be trying all sorts of default usernames trying to figure out how to make it work; and of course it is not configurable on the client side, it was also not super obvious in the milestone documentation.  You had to go through their support sites to figure it out in most cases.

Their documentation has gotten significantly better the last couple of years, and I finally decided to re-read the screen recorder install documentation to see what I might have missed since all the old tricks I had learned over the years were not working.  Low and behold, it seems as though they change the hard coded username from "Milestone" with a capital M, to "videoos", as described here.

I had to laugh when I found this change, things worked so smoothly after figuring that out.  But the change is amusing because I can think of no good reason for it.  It is going to irritate all their clients who learned the old way, it will not increase security because it is still a hard coded username like before, and you still can't alter it on the client machine.  I am happy they at least thought to document the unannounced change though as it has not yet made it to any support forum I have found.

Syncing a similar code base between multiple client instances

For years one of the companies I worked for has juggled multiple copies of a very similar code base.  The majority of the code was (or could be) identical, but there was just enough of a difference that a single code base could not be used.  An additional complicating factor was that the company was known by its clients for the ability to quickly turn around feature change requests; which meant that when one client wanted a change, there was not time to test its impact for all clients before rolling it out.

As you can imagine, this was a very difficult process to manage, and very time consuming re-creating identical features when other clients decided they wanted something that had been developed.

Over the years multiple attempts have been made to solve this issue.
- Branching was scrapped because changes can really only be pushed from the base branch to the child/client branches and there was not generally time to test all feature updates when a client wanted just one single change pushed into their code base.
- A shared service architecture was scrapped because versioning quickly became unwieldy between the clients using it, and the shared services started to become fractured.  They also suffered from the same inability to easily test and regression test all combinations of the endpoints.  Also a shared database became a security concern.
- A shared dll was scrapped for similar reasons when one client updated the dll, and the other clients were forced to take all the updates on their next modification.
- Splitting the code into multiple projects by major feature area.  This allowed for smaller pushes when changes were made, and attempts could be made to keep the most similar projects in sync, but it was still unwieldy.

As the features became more numerous regression testing became a big issue.  So a fairly comprehensive automated unit test and UI testing system was developed.  This significantly reduced the danger of moving features between the client code bases, but it did nothing to reduce the time involved.

A lot of posts were reviewed, and a lot of tools tried in an effort to figure out how to have a human easily view and push changes around between all the code bases.

- SyncBackPro (great sync tool, but no human review during the process)
- Winmerge
- Vim
- Diffuse (amazing tool, but crashes on windows with three or more files open)
- Code Compare (best code compare tool found, but only supported up to three files)

Over time Code Compare became the hands down favorite in the company for comparing code files, it was a smoother compare process for code files than any other tool tried; and made pushing changes around much easier.

However, even though it was smoother, it was still a massively huge process; and getting bigger as more and more features were added.  Others have had similar problems, but no one had any amazing and workable general solutions; although many people had tried playing with various git branching type features.  One company even developed a piece of software attempting to tackle this problem, however it appears to be primarily for UI development rather than backend code.

In this ever evolving situation the next attempt build on the popular Code Compare.  A custom powershell script was developed that mapped code bases to either other, or back to a master allowing the code to be instantly compared using Code Compare between a "master" instance of the code so the developer could push new features back to master, and pull down any desired feature differences.

The custom script:
- installs/removes itself in the right click menus for files and folders
- includes directions for installing itself in those menus inside of Visual Studio
- requires the code bases being compared to have identical directory and file structures
- requires that you go through and modify the $parentProjects variable to map your project folders
- uses the $parentProjects mapping to detect the incoming file/folder path, and open a comparison with Code Compare to the file/folder of the corresponding mapped path

Here is the Custom Power Shell Script

#
# VISUAL STUDIO INSTALLATION INSTRUCTIONS:
# Menu: Tools / External Tools
# - Click Add
# - Title: Mas Code Compare
# - Command: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
# - Arguments: -File "D:\Documents\scripts\CompareProjects.ps1" "$(ItemPath)"
# - memorize the index number of the item you just created (how far down in the list it is)
# Menu: Tools / Customize
# - Click the Commands tab
# - choose Context menu: Project and Solution Context Menus | Item
# - - Add Command...
# - - Choose the category Tools
# - - Select External Command
# - choose Context menu: Project and Solution Context Menus | Folder
# - - Add Command...
# - - Choose the category Tools
# - - Select External Command
#





if ($args[0] -eq $null) {
msg *, "Run with the following parameters: -install, -remove, 'PathOfFileOrFolder'"
return
}

#
# Install or Remove the windows context menu item
#
if ($args[0] -eq "-install" -or $args[0] -eq "-remove")
{
# AllFilesystemObjects is the key folder here, it specifies that the "shell" sub folder will be applied
# to all file system objects.
# the "shell" sub folder indicates that we are dealing with the right click context menu
# and the final folder name becomes the name of the menu item itself
$registryPath = "HKCR:\AllFilesystemObjects\shell\MasCodeCompare"
$regAutoPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Auto"
$regMasCodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.MasterCode"
$regCli1CodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Client1Code"
$regCli2CodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Client2Code"
New-PSDrive -Name HKCR -PSProvider Registry -Root HKEY_CLASSES_ROOT
New-PSDrive -Name HKLM -PSProvider Registry -Root HKEY_LOCAL_MACHINE

if ($args[0] -eq "-install")
{
# attempts to get command windows to not show with (/Q and -windowstyle hidden) don't work
$Name = "(Default)"
$value = "CMD.EXE /Q /C Powershell.exe -windowstyle hidden -File "+$PSScriptRoot.replace("\", "\\")+"\\CompareProjects.ps1 %1"

IF(!(Test-Path $registryPath))
{
New-Item -Path $registryPath -Force | Out-Null
New-Item -Path ($registryPath + "\command") -Force | Out-Null
New-Item -Path $regAutoPath -Force | Out-Null
New-Item -Path ($regAutoPath + "\command") -Force | Out-Null
New-Item -Path $regMasCodePath -Force | Out-Null
New-Item -Path ($regMasCodePath + "\command") -Force | Out-Null
New-Item -Path $regCli1CodePath -Force | Out-Null
New-Item -Path ($regCli1CodePath + "\command") -Force | Out-Null
New-Item -Path $regCli2CodePath -Force | Out-Null
New-Item -Path ($regCli2CodePath + "\command") -Force | Out-Null
}

New-ItemProperty -Path ($registryPath + "\command") -Name $name -Value $value -PropertyType String -Force | Out-Null
New-ItemProperty -Path $registryPath -Name "MUIVerb" -Value "Mas Code Compare" -PropertyType String -Force | Out-Null
New-ItemProperty -Path $registryPath -Name "SubCommands" -Value "MasCodeCompare.Auto;MasCodeCompare.MasterCode;MasCodeCompare.Client1Code;MasCodeCompare.Client2Code;" -PropertyType String -Force | Out-Null
New-ItemProperty -Path $regAutoPath -Name "MUIVerb" -Value "Auto choose" -PropertyType String -Force | Out-Null
New-ItemProperty -Path ($regAutoPath + "\command") -Name $name -Value $value -PropertyType String -Force | Out-Null
New-ItemProperty -Path $regMasCodePath -Name "MUIVerb" -Value "Master Code" -PropertyType String -Force | Out-Null
New-ItemProperty -Path ($regMasCodePath + "\command") -Name $name -Value ($value + " Company\MasterCode") -PropertyType String -Force | Out-Null
New-ItemProperty -Path $regCli1CodePath -Name "MUIVerb" -Value "Client 1" -PropertyType String -Force | Out-Null
New-ItemProperty -Path ($regCli1CodePath + "\command") -Name $name -Value ($value + " Client\Client1Code") -PropertyType String -Force | Out-Null
New-ItemProperty -Path $regCli2CodePath -Name "MUIVerb" -Value "Client 2" -PropertyType String -Force | Out-Null
New-ItemProperty -Path ($regCli2CodePath + "\command") -Name $name -Value ($value + " Client\Client2Code") -PropertyType String -Force | Out-Null
return
}
if ($args[0] -eq "-remove")
{
if (test-path $registryPath) { remove-item $registryPath -Recurse }
if (test-path $regAutoPath) { remove-item $regAutoPath -Recurse }
if (test-path $regMasCodePath) { remove-item $regMasCodePath -Recurse }
if (test-path $regCli1CodePath) { remove-item $regCli1CodePath -Recurse }
if (test-path $regCli2CodePath) { remove-item $regCli2CodePath -Recurse }
return
}
}

#
# If we have gotten to here, then we are probably trying to do a compare
#
$itemPath = $args[0]
$compareProject = $args[1]



$parentProjects = @{
"Client\Client2Code" = "Company\MasterCode";
"Client\Client1Code" = "Company\MasterCode";
"Company\MasterCode" = "Client\Client1Code"
}
$parentProject = $null

# find mapping that applies
foreach ($proj in $parentProjects.GetEnumerator())
{
if ($itemPath -like "*" + $proj.Name + "*") {
$parentProject = $proj
break
}
}

# if no parent found, alert user
if ($parentProject -eq $null)
{
Msg * "Invalid project selection"
return
}

# if a parent wasn't requested, then find parent from mapping
if ($compareProject -eq $null)
{
$compareProject = $parentProject.Value
}

$parentPath = $itemPath.replace($parentProject.Name, $compareProject)

& "C:\\Program Files\\Devart\\Code Compare\\CodeCompare.exe" "/environment=auto" "$parentPath" "$itemPath"

Free 000WebHosting Gotcha

I run a small family genealogy website for my immediate relatives.  It is not large, and generates very little traffic, so I have always tried to host it somewhere for free.  Most of it's life it lived on Google's free pages.  However, with Google's recent upgrade to their free web pages system, that system lost so much functionality that I was forced to look somewhere else for hosting.

After comparing multiple offerings I settled on 000webhost.com.  They had site limiters and other restrictions on their free offering, but again, all I need was a basic place for my personal family to be able to read a few stories.  They also allowed WordPress hosting for free, and I am pretty comfortable with WordPress, so it is my CMS of choice.

Everything went very well for multiple months.  We (my family) all worked hard and got the content manually copied from Google's old static page system into the new WordPress system hosted by 000webhost.  After the initial copy several months were spent adding new data, and fixing some of the formatting issues that occurred during migration.

During this time period I was creating backups once every couple of months, which is the only thing that ended up saving us.

One day I got an email from a family member saying that the site was telling her it had been deleted.  I went to login and sure enough, my cPanel login was gone, the entire website was just gone.

I did a little research in the 000webhost KB system, and found an article that said because it was free hosting, if your site ever got deleted, too bad so sad, it was just gone.  I could understand this, it was a free site, that policy made sense; I just didn't know why my site had been deleted.

After contacting 000webhost to see what had happened, I received a very polite email back saying that because I had not logged into the site in a long time, they had deleted it.  I was a bit taken aback, we had been actively working on the site every week for months.  With a bit more clarification I learned that in order for activity on the site to count, it had to be a login specifically to the cPanel system.

At this point I was finally irritated.  It was a free hosting service so any policy they wanted to put in place was fine with me, as long as they were up front about it.  And that was my problem, many other sites had been up front about having a policy like that, but 000webhost had not.  It might be buried somewhere on their site, but at the time I signed up with them there was no warning that this would occur.

This is where the politeness ended from 00webhost, after expressing my displeasure that my site had been deleted with no warning, and no initial knowledge on my part, the response from them said I should have gotten a warning email, but it didn't always work; and they seemed rather excited to see me go.

Needless to say, I took my most recent backup and went to find another host.  There's a decent chance we will start paying for a host eventually as the site grows, but I will never give 000webhost or their parent company hostinger.com my business after an experience like this.

Accord.net Machine Learning

Machine learning has been out of reach of the common developer for much of its early life.  While R has come along and stood out as "the" statistics language, it does not easily plug into the more mainstream languages.  Fortunately for us, in the last few years a library called accord-framework.net has grown up to fill this gap.

This framework is written in C# and allows the average .net developer access to a large number of machine intelligence algorithms that require very little statistical knowledge to actually use.  I specify the statistical qualifier because it does take a decent amount of basic c# experience to overcome some of the rough edges in validation the library still has.

The website also includes an impressive amount of documentation with code examples, unfortunately many of these examples are aging and have broken pieces in them due to changes in the software, however they are usually enough to get someone up and going after a little playing around.

Concepts and Implementation

The concept is fairly simple.  All the machine learning algorithms in this library take a two dimensional array of numbers (int or double), along with a single dimensional array of values with the correct outputs.  The algorithm then trains itself on these numbers.

After training you send it another set of numbers in the same format, and this time it will give you back what it thinks the outputs will be.

Your process will look something like this:
- Load datatable with data
- Codify the datatable into integer values using the Codification library
- Extract a two dimensional integer array using a combination of the datatable and the newly created code mapping object for all the columns you want to use as inputs.
- Extract a single dimensional integer array just like you did the two dimensional one, only this time for the single column that holds the values you are trying to guess.  It is important that the order of the values in this single column cause them to correspond with the correct input column values by index location.  If all the columns are being extracted from the same datatable then this should happen naturally.
- You will pass these two arrays into the desired algorithm to train it, some algorithms will require multiple training cycles to tune them.
- Once you have a trained algorithm object you can then pass it another two dimensional array of integers.  This time the values will be used to guess a single dimensional array of output integer values.  One gotcha here is that many of the algorithms can't handle input values they have not been trained on, so you can't throw just anything at it.

Because it works only with integers, any string values you want to use as input or output must first be converted to integers with no gaps between the number values.  You can do this on your own, but for convenience they have provide a special Code library which handles converting standard tables of data into encoded integers and back.

Gotchas

There are multiple bugs and missing features in this Code library, which is one of the biggest challenges that has to be overcome when working with these algorithms.  However, despite these issues I have still chosen to use the conversion library.

I have discovered that it's biggest shortcoming is that it is not capable of handling NULL values in the data.  So first you have to loop through every single value in your datatable and remove all NULL values.  From a speed perspective, this flaw alone probably means it would make for faster code to roll your own; but for the majority of developers out there, it is likely not worth the additional time to do that.

I have read that there is a default value setting inside the library at a per column level that allows the library to deal with NULLs.  For some reason that default either does not work, or is not initialized on its own.

The next annoying issue this Codification library has is really more of a versioning problem.  It looks like over time, rather than fixing particular issues, new methods get created to handle the new cases.  So you end up with multiple methods that run off different logic when encoding values.  Specifically, there seems to be a big difference in Codification between the Transform method and the Apply method.  Transform seems to attempt to specifically encode all columns requested, kind of like the explicit class creation overload that accepts a list of columns.  Apply on the other hand logically processes a datatable detecting which columns need to be converted and which do not.

Thoughts and Concepts

Most of my work in this area has been with the various Classification algorithms in the accord.net library.  These seem to have the limitation of not being able to accept a Continuous (un-encoded int) value type as the output column it is supposed to be guessing.  The solution to this particular issue is probably to switch to using a Regression algorithm.

Something else that might not immediately come to the mind of the average developer is that the output is not going to have any human readable meaning.  Because the system works exclusively with integer arrays, the output will just be an array of numbers.  These output numbers must then be passed through the Codification library a second time, this time in reverse, to get back to the human readable version of them.