Syncing a similar code base between multiple client instances

For years one of the companies I worked for has juggled multiple copies of a very similar code base.  The majority of the code was (or could be) identical, but there was just enough of a difference that a single code base could not be used.  An additional complicating factor was that the company was known by its clients for the ability to quickly turn around feature change requests; which meant that when one client wanted a change, there was not time to test its impact for all clients before rolling it out.

As you can imagine, this was a very difficult process to manage, and very time consuming re-creating identical features when other clients decided they wanted something that had been developed.

Over the years multiple attempts have been made to solve this issue.
- Branching was scrapped because changes can really only be pushed from the base branch to the child/client branches and there was not generally time to test all feature updates when a client wanted just one single change pushed into their code base.
- A shared service architecture was scrapped because versioning quickly became unwieldy between the clients using it, and the shared services started to become fractured.  They also suffered from the same inability to easily test and regression test all combinations of the endpoints.  Also a shared database became a security concern.
- A shared dll was scrapped for similar reasons when one client updated the dll, and the other clients were forced to take all the updates on their next modification.
- Splitting the code into multiple projects by major feature area.  This allowed for smaller pushes when changes were made, and attempts could be made to keep the most similar projects in sync, but it was still unwieldy.

As the features became more numerous regression testing became a big issue.  So a fairly comprehensive automated unit test and UI testing system was developed.  This significantly reduced the danger of moving features between the client code bases, but it did nothing to reduce the time involved.

A lot of posts were reviewed, and a lot of tools tried in an effort to figure out how to have a human easily view and push changes around between all the code bases.

- SyncBackPro (great sync tool, but no human review during the process)
- Winmerge
- Vim
- Diffuse (amazing tool, but crashes on windows with three or more files open)
- Code Compare (best code compare tool found, but only supported up to three files)

Over time Code Compare became the hands down favorite in the company for comparing code files, it was a smoother compare process for code files than any other tool tried; and made pushing changes around much easier.

However, even though it was smoother, it was still a massively huge process; and getting bigger as more and more features were added.  Others have had similar problems, but no one had any amazing and workable general solutions; although many people had tried playing with various git branching type features.  One company even developed a piece of software attempting to tackle this problem, however it appears to be primarily for UI development rather than backend code.

In this ever evolving situation the next attempt build on the popular Code Compare.  A custom powershell script was developed that mapped code bases to either other, or back to a master allowing the code to be instantly compared using Code Compare between a "master" instance of the code so the developer could push new features back to master, and pull down any desired feature differences.

The custom script:
- installs/removes itself in the right click menus for files and folders
- includes directions for installing itself in those menus inside of Visual Studio
- requires the code bases being compared to have identical directory and file structures
- requires that you go through and modify the $parentProjects variable to map your project folders
- uses the $parentProjects mapping to detect the incoming file/folder path, and open a comparison with Code Compare to the file/folder of the corresponding mapped path

Here is the Custom Power Shell Script

#
# VISUAL STUDIO INSTALLATION INSTRUCTIONS:
# Menu: Tools / External Tools
# - Click Add
# - Title: Mas Code Compare
# - Command: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
# - Arguments: -File "D:\Documents\scripts\CompareProjects.ps1" "$(ItemPath)"
# - memorize the index number of the item you just created (how far down in the list it is)
# Menu: Tools / Customize
# - Click the Commands tab
# - choose Context menu: Project and Solution Context Menus | Item
# - - Add Command...
# - - Choose the category Tools
# - - Select External Command
# - choose Context menu: Project and Solution Context Menus | Folder
# - - Add Command...
# - - Choose the category Tools
# - - Select External Command
#





if ($args[0] -eq $null) {
msg *, "Run with the following parameters: -install, -remove, 'PathOfFileOrFolder'"
return
}

#
# Install or Remove the windows context menu item
#
if ($args[0] -eq "-install" -or $args[0] -eq "-remove")
{
# AllFilesystemObjects is the key folder here, it specifies that the "shell" sub folder will be applied
# to all file system objects.
# the "shell" sub folder indicates that we are dealing with the right click context menu
# and the final folder name becomes the name of the menu item itself
$registryPath = "HKCR:\AllFilesystemObjects\shell\MasCodeCompare"
$regAutoPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Auto"
$regMasCodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.MasterCode"
$regCli1CodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Client1Code"
$regCli2CodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Client2Code"
New-PSDrive -Name HKCR -PSProvider Registry -Root HKEY_CLASSES_ROOT
New-PSDrive -Name HKLM -PSProvider Registry -Root HKEY_LOCAL_MACHINE

if ($args[0] -eq "-install")
{
# attempts to get command windows to not show with (/Q and -windowstyle hidden) don't work
$Name = "(Default)"
$value = "CMD.EXE /Q /C Powershell.exe -windowstyle hidden -File "+$PSScriptRoot.replace("\", "\\")+"\\CompareProjects.ps1 %1"

IF(!(Test-Path $registryPath))
{
New-Item -Path $registryPath -Force | Out-Null
New-Item -Path ($registryPath + "\command") -Force | Out-Null
New-Item -Path $regAutoPath -Force | Out-Null
New-Item -Path ($regAutoPath + "\command") -Force | Out-Null
New-Item -Path $regMasCodePath -Force | Out-Null
New-Item -Path ($regMasCodePath + "\command") -Force | Out-Null
New-Item -Path $regCli1CodePath -Force | Out-Null
New-Item -Path ($regCli1CodePath + "\command") -Force | Out-Null
New-Item -Path $regCli2CodePath -Force | Out-Null
New-Item -Path ($regCli2CodePath + "\command") -Force | Out-Null
}

New-ItemProperty -Path ($registryPath + "\command") -Name $name -Value $value -PropertyType String -Force | Out-Null
New-ItemProperty -Path $registryPath -Name "MUIVerb" -Value "Mas Code Compare" -PropertyType String -Force | Out-Null
New-ItemProperty -Path $registryPath -Name "SubCommands" -Value "MasCodeCompare.Auto;MasCodeCompare.MasterCode;MasCodeCompare.Client1Code;MasCodeCompare.Client2Code;" -PropertyType String -Force | Out-Null
New-ItemProperty -Path $regAutoPath -Name "MUIVerb" -Value "Auto choose" -PropertyType String -Force | Out-Null
New-ItemProperty -Path ($regAutoPath + "\command") -Name $name -Value $value -PropertyType String -Force | Out-Null
New-ItemProperty -Path $regMasCodePath -Name "MUIVerb" -Value "Master Code" -PropertyType String -Force | Out-Null
New-ItemProperty -Path ($regMasCodePath + "\command") -Name $name -Value ($value + " Company\MasterCode") -PropertyType String -Force | Out-Null
New-ItemProperty -Path $regCli1CodePath -Name "MUIVerb" -Value "Client 1" -PropertyType String -Force | Out-Null
New-ItemProperty -Path ($regCli1CodePath + "\command") -Name $name -Value ($value + " Client\Client1Code") -PropertyType String -Force | Out-Null
New-ItemProperty -Path $regCli2CodePath -Name "MUIVerb" -Value "Client 2" -PropertyType String -Force | Out-Null
New-ItemProperty -Path ($regCli2CodePath + "\command") -Name $name -Value ($value + " Client\Client2Code") -PropertyType String -Force | Out-Null
return
}
if ($args[0] -eq "-remove")
{
if (test-path $registryPath) { remove-item $registryPath -Recurse }
if (test-path $regAutoPath) { remove-item $regAutoPath -Recurse }
if (test-path $regMasCodePath) { remove-item $regMasCodePath -Recurse }
if (test-path $regCli1CodePath) { remove-item $regCli1CodePath -Recurse }
if (test-path $regCli2CodePath) { remove-item $regCli2CodePath -Recurse }
return
}
}

#
# If we have gotten to here, then we are probably trying to do a compare
#
$itemPath = $args[0]
$compareProject = $args[1]



$parentProjects = @{
"Client\Client2Code" = "Company\MasterCode";
"Client\Client1Code" = "Company\MasterCode";
"Company\MasterCode" = "Client\Client1Code"
}
$parentProject = $null

# find mapping that applies
foreach ($proj in $parentProjects.GetEnumerator())
{
if ($itemPath -like "*" + $proj.Name + "*") {
$parentProject = $proj
break
}
}

# if no parent found, alert user
if ($parentProject -eq $null)
{
Msg * "Invalid project selection"
return
}

# if a parent wasn't requested, then find parent from mapping
if ($compareProject -eq $null)
{
$compareProject = $parentProject.Value
}

$parentPath = $itemPath.replace($parentProject.Name, $compareProject)

& "C:\\Program Files\\Devart\\Code Compare\\CodeCompare.exe" "/environment=auto" "$parentPath" "$itemPath"

Free 000WebHosting Gotcha

I run a small family genealogy website for my immediate relatives.  It is not large, and generates very little traffic, so I have always tried to host it somewhere for free.  Most of it's life it lived on Google's free pages.  However, with Google's recent upgrade to their free web pages system, that system lost so much functionality that I was forced to look somewhere else for hosting.

After comparing multiple offerings I settled on 000webhost.com.  They had site limiters and other restrictions on their free offering, but again, all I need was a basic place for my personal family to be able to read a few stories.  They also allowed WordPress hosting for free, and I am pretty comfortable with WordPress, so it is my CMS of choice.

Everything went very well for multiple months.  We (my family) all worked hard and got the content manually copied from Google's old static page system into the new WordPress system hosted by 000webhost.  After the initial copy several months were spent adding new data, and fixing some of the formatting issues that occurred during migration.

During this time period I was creating backups once every couple of months, which is the only thing that ended up saving us.

One day I got an email from a family member saying that the site was telling her it had been deleted.  I went to login and sure enough, my cPanel login was gone, the entire website was just gone.

I did a little research in the 000webhost KB system, and found an article that said because it was free hosting, if your site ever got deleted, too bad so sad, it was just gone.  I could understand this, it was a free site, that policy made sense; I just didn't know why my site had been deleted.

After contacting 000webhost to see what had happened, I received a very polite email back saying that because I had not logged into the site in a long time, they had deleted it.  I was a bit taken aback, we had been actively working on the site every week for months.  With a bit more clarification I learned that in order for activity on the site to count, it had to be a login specifically to the cPanel system.

At this point I was finally irritated.  It was a free hosting service so any policy they wanted to put in place was fine with me, as long as they were up front about it.  And that was my problem, many other sites had been up front about having a policy like that, but 000webhost had not.  It might be buried somewhere on their site, but at the time I signed up with them there was no warning that this would occur.

This is where the politeness ended from 00webhost, after expressing my displeasure that my site had been deleted with no warning, and no initial knowledge on my part, the response from them said I should have gotten a warning email, but it didn't always work; and they seemed rather excited to see me go.

Needless to say, I took my most recent backup and went to find another host.  There's a decent chance we will start paying for a host eventually as the site grows, but I will never give 000webhost or their parent company hostinger.com my business after an experience like this.

Accord.net Machine Learning

Machine learning has been out of reach of the common developer for much of its early life.  While R has come along and stood out as "the" statistics language, it does not easily plug into the more mainstream languages.  Fortunately for us, in the last few years a library called accord-framework.net has grown up to fill this gap.

This framework is written in C# and allows the average .net developer access to a large number of machine intelligence algorithms that require very little statistical knowledge to actually use.  I specify the statistical qualifier because it does take a decent amount of basic c# experience to overcome some of the rough edges in validation the library still has.

The website also includes an impressive amount of documentation with code examples, unfortunately many of these examples are aging and have broken pieces in them due to changes in the software, however they are usually enough to get someone up and going after a little playing around.

Concepts and Implementation

The concept is fairly simple.  All the machine learning algorithms in this library take a two dimensional array of numbers (int or double), along with a single dimensional array of values with the correct outputs.  The algorithm then trains itself on these numbers.

After training you send it another set of numbers in the same format, and this time it will give you back what it thinks the outputs will be.

Your process will look something like this:
- Load datatable with data
- Codify the datatable into integer values using the Codification library
- Extract a two dimensional integer array using a combination of the datatable and the newly created code mapping object for all the columns you want to use as inputs.
- Extract a single dimensional integer array just like you did the two dimensional one, only this time for the single column that holds the values you are trying to guess.  It is important that the order of the values in this single column cause them to correspond with the correct input column values by index location.  If all the columns are being extracted from the same datatable then this should happen naturally.
- You will pass these two arrays into the desired algorithm to train it, some algorithms will require multiple training cycles to tune them.
- Once you have a trained algorithm object you can then pass it another two dimensional array of integers.  This time the values will be used to guess a single dimensional array of output integer values.  One gotcha here is that many of the algorithms can't handle input values they have not been trained on, so you can't throw just anything at it.

Because it works only with integers, any string values you want to use as input or output must first be converted to integers with no gaps between the number values.  You can do this on your own, but for convenience they have provide a special Code library which handles converting standard tables of data into encoded integers and back.

Gotchas

There are multiple bugs and missing features in this Code library, which is one of the biggest challenges that has to be overcome when working with these algorithms.  However, despite these issues I have still chosen to use the conversion library.

I have discovered that it's biggest shortcoming is that it is not capable of handling NULL values in the data.  So first you have to loop through every single value in your datatable and remove all NULL values.  From a speed perspective, this flaw alone probably means it would make for faster code to roll your own; but for the majority of developers out there, it is likely not worth the additional time to do that.

I have read that there is a default value setting inside the library at a per column level that allows the library to deal with NULLs.  For some reason that default either does not work, or is not initialized on its own.

The next annoying issue this Codification library has is really more of a versioning problem.  It looks like over time, rather than fixing particular issues, new methods get created to handle the new cases.  So you end up with multiple methods that run off different logic when encoding values.  Specifically, there seems to be a big difference in Codification between the Transform method and the Apply method.  Transform seems to attempt to specifically encode all columns requested, kind of like the explicit class creation overload that accepts a list of columns.  Apply on the other hand logically processes a datatable detecting which columns need to be converted and which do not.

Thoughts and Concepts

Most of my work in this area has been with the various Classification algorithms in the accord.net library.  These seem to have the limitation of not being able to accept a Continuous (un-encoded int) value type as the output column it is supposed to be guessing.  The solution to this particular issue is probably to switch to using a Regression algorithm.

Something else that might not immediately come to the mind of the average developer is that the output is not going to have any human readable meaning.  Because the system works exclusively with integer arrays, the output will just be an array of numbers.  These output numbers must then be passed through the Codification library a second time, this time in reverse, to get back to the human readable version of them.

Microsoft Owin Http append header bug

I've been dealing with an exception for most of this year that has really been annoying.

This bug is specific to .net Owin code, possibly even the CookieAuthentication method when running under IIS in an integrated pipeline.  While this sounds very specific, I believe it is actually one of the more common setups for .net and IIS.

After the first couple of months I looked into it and found that others were having the same issue.  Unfortunately, Microsoft already knew about it and responded that the fix for it was such a low priority that they were not going to bother with it any time soon.

After many months of dealing with this annoying bug I finally found time to spend several days digging into Microsoft's code to see what was causing it.

As the Microsoft guy had indicated, it was the parent/child request scenario in IIS that was triggering the issue.  When I first read his response I did not really understand what he meant by it, or how it was triggering the errors.

Recreating the error

I needed to re-create it for myself so I could understand what was going on.  I started by setting my sign-in cookie to expire after 10 seconds on my login page.

            var authenticationManager = HttpContext.Current.GetOwinContext().Authentication;
            authenticationManager.SignIn(new AuthenticationProperties()
            {
                IsPersistent = chkRemember.Checked, // Tells OWIN whether or not to create a persistent cookie instead of just a session cookie
                IssuedUtc = DateTime.UtcNow,
                ExpiresUtc = DateTime.UtcNow.AddSeconds(10)
            }, userIdentity);

Next, in my Startup.cs I set my ExpireTimeSpan to 1 minute, and my validateInterval to 10 seconds.  I'm not sure that ExpireTimeSpan actually does anything, it feels like it gets over-ridden by the ExpiresUtc from the SignIn code.

            var cookieExpiration = TimeSpan.FromMinutes(1);
            var checkPeriod = TimeSpan.FromSeconds(10);

            app.UseCookieAuthentication(new CookieAuthenticationOptions
            {
                ...
                Provider = new CookieAuthenticationProvider
                {
                    OnValidateIdentity = SecurityStampValidator.OnValidateIdentity(
                        validateInterval: checkPeriod,
                        regenerateIdentity: (manager, user) => manager.CreateIdentityAsync(user, DefaultAuthenticationTypes.ApplicationCookie)
                        ),
                },
                ExpireTimeSpan = cookieExpiration,
                SlidingExpiration = true,
            });

Once I had these very short times in place I started up the site.  By first logging in and then, after 5 seconds had passed and before the 10 second mark, navigating to the root of the site ( "/" ) I was able to consistently generate the error.  Every 5 seconds or so if I went to the root I would throw that error.  As expected, the user never actually sees the error, it just gets thrown in back end code, so any logging you have setup should catch it.

Cause

If you did not read any of the articles linked to above, this error is caused by two things.  First, IIS in integrated mode will execute the .net code twice when the site root is requested.  First it starts what is called the "parent" call, but immediately sees that the request is for root, which does not exist, so it starts a "child" call to whatever the default document is.  This child call executes correctly and is prepped for return to the client.  When the child call is finished, the parent call resumes executing, only now the Response has already been created and prepared for the client, and this is where the Owin bugs come into play.

Because the Owin code does not check before attempting to set header information in the default ChunkingCookieManager, it blows up during this parent call as Owin is trying to write out cookie information to the header.  This bug actually exists at least twice in Owin, so if you fix the first one, then the second one still blows up with the same error just in a different place.

The bad news is that there is no good or easy way to deal with this error.  The good news is that there is a way of wrapping the first error so that your error handler can uniquely identify and handle it with minimal effort.

Solution 1:

In Global.asax.cs in the Application_Error method, where you are likely trapping the errors, you can inspect the error description as well as its stack trace and make a decision on how to handle it that way.

        protected void Application_Error(object sender, EventArgs e)

            Exception parent_ex = Server.GetLastError();

            Exception ex = parent_ex.GetBaseException();

            if(ex.Message.StartsWith("Server cannot append header after HTTP headers have been sent.") && ex.StackTrace.Contains("Microsoft.Owin.Infrastructure.ChunkingCookieManager.AppendResponseCookie"))  then do something

Solution 2:

While neither of these solutions is great, and solution 1 is easier, it also feels messier.  The second solution is to wrap the CookieManager and catch and wrap the exception there.  This feels cleaner because it is much closer to the source, unfortunately you cannot handle the error there or the next error will just crop up.

replace the default CookieManager with your custom wrapper:

            app.UseCookieAuthentication(new CookieAuthenticationOptions
            {
                ...
                CookieManager = new ChunkingCookieManagerHeaderSentCheck(),
                ...

then, for the most part, just wrap the default ChunkingCookieManager.  However, in the problem method insert a try/catch that provides a bit more detail.  You still have to look for this detail in Global.asax.cs, but it is better than trying to comb through a stack trace.  You could also use a custom exception here with extra fields that could help you in identifying this specific exception.

    public class ChunkingCookieManagerHeaderSentCheck : ICookieManager

    {

        private readonly ChunkingCookieManager _chunkingCookieManager;

        public ChunkingCookieManagerHeaderSentCheck()

        {

            _chunkingCookieManager = new ChunkingCookieManager();

        }

        public string GetRequestCookie(IOwinContext context, string key)

        {

            return _chunkingCookieManager.GetRequestCookie(context, key);

        }

        public void AppendResponseCookie(IOwinContext context, string key, string value, CookieOptions options)

        {

            /*

             * Microsof's Chunking Cookie Manager has a bug in it where it does not check to see

             * if headers have already been sent before it attempts to set more.

             * The exception cannot be dealt with here because the same bug exists in

             * CookieAuthenticationHandler.ApplyResponseGrantAsync.

             * Since it is harder to catch that one, it is best to just let this one be thrown,

             * but add some explanation to it so error handling can single it out in Global.asax.cs

             * and deal with it appropriately.

             */

            try

            {

                _chunkingCookieManager.AppendResponseCookie(context, key, value, options);

            }

            catch (Exception ex)

            {

                if (context.Request.Uri.AbsolutePath == "/" && ex.Message.StartsWith("Server cannot append header after HTTP headers have been sent"))

                    throw new Exception("Owin CookieManager attempted to write headers to root response.", ex);

                else

                    throw;

            }

        }

        public void DeleteCookie(IOwinContext context, string key, CookieOptions options)

        {

            _chunkingCookieManager.DeleteCookie(context, key, options);

        }

    }

Conclusion

Hopefully Microsoft will fix their code some day.  However it has been a known issue since June 2017, and so far they have only made excuses for why they do not feel like fixing it.  So until it actually makes it onto their road map, these appear to be the best options for dealing with the issue.

As a quick note.  I read that Server.Transfer and Server.Execute also trigger this parent/child request feature of IIS.  If that is true, then they should also trigger this bug.  However, in my testing with Server.Transfer, I was not able to trip this bug.

OWASP ZAP API

When testing a web application there are many different types of tests you can run and automate, and many different ways of doing so.  I currently work in an environment where Microsoft's Team Foundation Server (now called Azure DevOps Server) runs a series of Unit tests and Selenium tests in an automated fashion.  We initially used CodedUI until Microsoft discontinued it, then we converted our code base.

This setup is one giant collection of regression tests like every project should have.  And mingled in with them was some basic security testing from a users perspective as well.  However, what we did not have was a method of automating the ability to test hack attempts against our sites.  One of the reasons why is the shear number of possible hacks and combination hacks that are possible and should be tested for.

Enter the OWASP ZAP tool.  A bunch of security experts have formed a non profit to educate people about security, and one of that companies outputs has been a free (and well maintained) tool for attacking your own sites and producing reports about any security issues it finds.  The best part is that this tool has a pretty well developed REST API, so you can run it in an automated fashion.

The ZAP tool has a decent API UI to help you learn it, but other than that is lacking documentation for it, which is the reason for this post.

If you open the ZAP tool GUI, click on Tools / Options / API you can find your API key and a few API related settings you can mess with.  By default, the tool has a web UI at http://localhost:8080/.  Clicking on the Local API link inside of that will get you into the REST API help and demo area where you can run each call.

In the GUI you can enter the base URL for your site, and click Attack.  That is an easy method of running a one-off attack against your site.  But if you want to automate this process, there are several REST calls involved.

The first thing you need to do is start the application via command line.  On windows it installs to:
C:\Program Files\OWASP\Zed Attack Proxy\zap.bat

Then call these REST endpoints:
- NewSession (clear all unsaved scan histories, just for a fresh start; you could load saved sessions instead if desired)
http://localhost:8080/JSON/core/action/newSession/?zapapiformat=JSON&apikey=[yourapikey]
- SetMode ( sets the scan mode to attack level )
http://localhost:8080/JSON/core/action/setMode/?zapapiformat=JSON&apikey=[yourapikey]&mode=attack
- Spider.Scan ( spider the url to find all the links to attack, this has to be performed before an Ascan attack is performed, the attack itself is only done on URLs already in the ZAP session.  The Spider is the most efficient method of crawling one or more pages and loading all the found URLs into memory.  The attack method only attacks distinct URLs, so it does not matter if the spider duplicates URLs while it is crawling pages. )
http://localhost:8080/JSON/spider/action/scan/?zapapiformat=JSON&apikey=[yourapikey]&url=[an escaped url to crawl]&recurse=true
- AScan.Scan ( fire off an attack for all links the spider found under this base URL )
http://localhost:8080/JSON/ascan/action/scan/?zapapiformat=JSON&apikey=[yourapikey]&url=[an escaped url to lookup]&recurse=true
- JsonReport ( get the total report for all scans and attacks in json format )
http://localhost:8080/OTHER/core/other/jsonreport/?apikey=[yourapikey]

There are many more options, and other REST endpoints you can call to customize from there and to perform other and more detailed automated attacks.  However, those are the base required to create an automated scan and attack.  Hopefully that provides enough of a jump start so others can get the concept down and process further on their own.

Installing Milestone Xprotect +

I have been a long time fan of the Milestone XProtect NVR software.  I have multiple installations, and have fought through many install and upgrade issues.

Before the Plus line of their software came out the XProtect product was pretty stable.  It just worked, albeit a bit limited in features.  The new Plus line fixes all the feature issues (lots of fun new features), but the install and upgrade process has become extremely fragile in the process.  Hopefully over time they will start to fix some of these issues.

In the meantime, I thought I would create an install list for anyone else struggling with it.  Almost all of my installs are on smaller windows 10 servers.  Considering that is not even a server OS, it was an afterthought by Milestone when they first started releasing the new product line, and might be the cause of some of these problems.

Install Steps:
- Uninstall IIS (Milestone will automatically install and configure IIS, if you already have it installed then sometimes the install will fail, especially if you have changed any of the default settings).
- Install SQL 2016 Express.  Milestone does not yet support later versions of SQL, and SQL 2015 that it ships with has a random bug in the GEO portion of its code that can cause the Milestone installation to fail.  While technically a SQL bug, Milestone could easily fix this by simply shipping sql 2016 with their product instead of 2015.
( I have been informed by Milestone that as of version 2018, SQL 2016 is the default shipped with it. )
- Do not at any point re-name the computer this is being installed on.  There is a Milestone/SQL bug where Milestone will get the old computer name from SQL and fail during install if you do this.  Again, a SQL bug, but one that Milestone could solve.  If you really want to rename it, then I have heard that there is a SQL script you can run so SQL sees the new computer name.  Research renaming a sql server computer name.
- Next start the Milestone installation, however, do a custom installation and only install the Recording server (assuming you are planning on an all-in-one box).  If you try and install all the software at once then it generally fails.  I believe this is due to a security bug in the Mobile Server install, but am not sure.
- Now do another Custom Milestone install this time installing everything.  Again, the custom install will prevent the Mobile server from being automatically installed.
- Open up the Management Client and create a new local admin account.  XProtect has some security bugs I have not yet figured out related to using the build in windows accounts which is the default.
- Use this local admin account when connecting to the Smart Client if the build-in account does not work.  Not a bad habit anyway in case you log into the computer with a different account in the future.
- Find and install the Milestone Mobile Server package.  When it asks for credentials to connect to the recording server, use the new local admin account you created instead of the default of the windows credentials.

At this point you should be up and running with a new install of Milestone XProtect+.  I have run into issues during upgrades where the database got corrupted in some fashion so everything looked good, but then I started having quirky issues when messing around with trying to bring cameras back online.

Pacific Power Hourly Energy Savings

On the North West coast of the United States is a company called Pacific Power.  They are the government approved power provider.

There is a special program they have that purports to incentivize people to shift their power usage to off peak periods.  This is a very good idea to help balance out the power grid, and I am a huge fan of the idea.  However, it does cause a great deal of work for those who shift their usage and they do deserve to be incentivized to do so.

Unfortunately, it turns out that the program Pacific Power runs is a scam.  Their documentation, and their representatives who try and explain the documentation, claims that the normal household is charged 3 cents per kilowatt hour.  Once switched to the new program the household will be charged 6 cents per kilowatt hour for on peak usage, and only 1 cent per kilowatt hour for off peak usage.  This seems reasonable, and a hard working person should be able to save some money by changing their habits fairly drastically.
( One of the documents actually says it will be a negative 1 cent, the reps just seem to explain it poorly )

The reality of this program is far different from what is advertised.

In reality, Pacific Power does not stop charging the household for anything, they continue charging the standard electrical fees, and then they tack on an ADDITIONAL 6 cents per kilowatt hour for peak usage, and a -1 cent per kilowatt hour for off peak usage.

So, by completely revamping your lifestyle, you might save $1-2 a month if you are lucky, if you slip then you will suddenly be paying a massively inflated bill on top of the normal electrical bill you were already paying.