tag:blogger.com,1999:blog-7031011612810724622024-03-13T23:19:11.894-07:00Development Memoirs<a href="http://www.linkedin.com/in/jeremeguenther"><img src="http://www.linkedin.com/img/webpromo/btn_viewmy_160x25.gif" width="160" height="25" border="0" alt="View Jereme Guenther's profile on LinkedIn"></a><br>
A collection of problems and discoveries usually related to the tech industry in some way.Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.comBlogger167125tag:blogger.com,1999:blog-703101161281072462.post-21298633138250696942023-03-21T09:44:00.001-07:002024-03-13T07:39:04.360-07:00EMP Shield Review<h2 style="text-align: left;">EMP and Lightening Protection</h2><p>As a quick disclaimer, I am not an electrician, however I am a software engineer and have been tinkering with things my entire life. So I am reasonably well versed in figuring out how things work.</p><p>With various political tensions around the world, and the ever present threat of another Carrington event, I have been interested in some way to protect myself from the effects of various sizes of EMP events. To that end I have been doing a good bit of research into what an EMP is, how it affects various things, and how to best protect against it.</p><p>The challenge is that not all EMPs are created equal, so we have to start with what it is, and what it does. An electromagnetic wave is essentially just a radio signal, a very powerful one. As such the frequency of the wave determines how it will affect things. The length of the antenna also plays a large role in picking up such a wave and generating electrical current from it.</p><h4 style="text-align: left;">EMP Components</h4><p>A nuclear EMP is made up of 3 major categories: the E1, E2, and E3 events. They are characterized by wavelength and period after detonation that they occur. The E1 event is a very high frequency event, my guess from <a href="https://www.wbdg.org/resources/high-altitude-emp-effects-protection">the chart here</a> and from an EMP Shield interview is that it will be in the gamma spectrum, despite the paper the chart is embedded in indicating it will only be a few hundred megahertz. The E2 event appears to be a lower frequency event, yet still in the gamma spectrum according to the same chart. The E3 Event is a low frequency, <a href="https://www.hsgac.senate.gov/wp-content/uploads/imo/media/doc/Testimony-Horton-2019-02-27.pdf">below 1 Hz</a>, starting at about the 1 second mark and can last for several minutes.</p><p>A geomagnetic storm only has the E3 component, and while it is generally estimated that it will not be as strong as the E3 event from a nuclear weapon, it will cover and affect a vastly larger geographic region.</p><p>Like all electromagnetic waves, including radio waves, an EMP is limited to line of site, or any objects the waves are able to penetrate.</p><p>So what does frequency and wavelength mean to us. Let's start with a simple electromagnetic <a href="https://qph.cf2.quoracdn.net/main-qimg-08142b53efe9a4d2568b8f1ded17f7f8">spectrum chart</a> someone was nice enough to <a href="https://www.quora.com/What-is-the-wavelength-of-gamma-rays-in-meters">upload to quora</a>. You can see on the left hand side (long wave side) we have radio waves, then microwaves and light, and gamma rays on the far right (short wave side). The longer the wavelength the longer than antenna required to generate a meaningful amount of current from the wave.</p><p>Most of us are familiar with car radios, truckers CBs, and walkie talkies. Those all have relatively long antennas and fall on the far left of this chart. Cell phones have much shorter antennas built into the body of the phones these days, they use the microwave spectrum. So you can imagine that super high frequency waves up in the gamma region are going to bind to every tiny little antenna they can find, which pretty much means every circuit on a circuit board.</p><p>The caveat to this is that now we enter the realm of speculation. Just how much current could a small wire generate before it got big enough to fry anything? And just how much would such a wave have to be reduced to in order to protect against that? There are examples on the Internet of people building small EMP devices that are capable of disrupting small electronics, but not frying them; and this is an important distinction when you're talking about the level of damage done. Clearly unexpected current introduced into a modern electronic device is going to cause it to miss behave, usually a reboot fixes the problem. What is not clear is how much current it would take to actually burn out a circuit, or if the wires on circuit boards are even capable of generating enough current to do that.</p><p>Once we get down to the E3 event or a geomagnetic storm we can see that the frequency of 1 Hz is super low. To get an idea of just how low look at this chart of standard <a href="https://hamradioprepper.com/us-amateur-radio-bands/">Ham Radio frequencies</a>. The Meter values next to the frequencies are the length of one full wave, and how long an optimal receive antenna would be to generate maximum voltage. Most of those frequencies are measured in Megahertz, and the conversion by a factor of 1000 goes Hertz, Kilohertz, Megahertz.</p><p>So we're talking an extremely long wavelength. What this means is it is unlikely to bind to anything smaller than an electrical line; which would in turn generate a massive surge into your house and fry everything plugged in.</p><h4 style="text-align: left;">EMP Protection</h4><p>Most of us have heard of <a href="https://commonsensehome.com/electromagnetic-pulse-emp/">EMP hardening</a> by putting things into a solid metal box. Or some sort of wire mesh faraday cage. It's important to note that such devices do not inherently block the EMP waves, they simply reduce them, if they get reduced to zero then they are effectively blocked. The thicker the metal box the more reduction happens.</p><p>Living inside a metal box is not only not practical for most people, it also does not fully solve the problem as the metal box can become charged and transfer the charge to things inside of it. So it is important for larger boxes to be grounded to allow that charge to flow to the ground.</p><p>And this is where logic starts to play a role. There is very little evidence we can pull from as to what an EMP would really do. The Carrington event is one of the most recent solar events we can look at. And there was some nuclear testing that generated EMPs we can draw data from to speculate on. It is also similar in effect to a lightening strike, so we can draw some data from lightening events as well.</p><p>Looking at all these data sources, it would seem that the highest risk items are long metal objects such as power lines that will provide great antennas to collect the energy and transform it into a power surge. Obviously the closer you are to the event the smaller the antennas that will be affected. And the easiest solution is if you could ground everything to get rid of the energy fast.</p><p>There are two problems with the grounding idea. Not all devices can be practically grounded, and most surge protectors do not react fast enough to stop such a surge. For reference a lightening strike lasts for around 1 second I believe, while the first part of an EMP surge, called an E1 event lasts for 1 nanosecond. The breakers in a standard breaker panel do not even react fast enough to fully protect against a lightening strike.</p><p><br /></p><h2 style="text-align: left;">Whole Home Surge Protectors</h2><p>In about 2021 electrical code change requiring something called a <a href="https://www.reddit.com/r/askanelectrician/comments/10w5k5c/comment/j81m9ck/?context=3">whole home surge protector</a>. These are devices that react in about 2 nanoseconds shunting electrical surges and giving the breakers time to trip. These devices can not handle current for very long due to their tiny 10 guage wires, but it is long enough for the breaker to trip. While these devices can handle smaller surges, they sacrifice themselves if they end up protecting the house from a lightening strike.</p><p>There are multiple well known and trusted whole house surge protectors, a couple of them are:<br /><a href="https://www.amazon.com/Siemens-FS140-Whole-House-Protection/dp/B013WINMK6/ref=sr_1_3_mod_primary_new?crid=RORGN56LICZQ&keywords=FS140%2BSurge%2BProtector&qid=1676314080&sbo=RZvfv%2F%2FHxDF%2BO5021pAnSA%3D%3D&sprefix=fs140%2Bsurge%2Bprotector%2Caps%2C240&sr=8-3&th=1">Siemens FS140</a> it sounds like this is the <a href="https://www.youtube.com/watch?v=0dZIYYR7iIs">highest rated one</a> on the market.<br /><a href="https://www.amazon.com/Eaton-CHSPT2SURGE-Whole-Protector-Single/dp/B01LXRNOEI/ref=sr_1_1_sspa?keywords=whole+house+surge+protector&qid=1676230370&sprefix=whole+house+surge%2Caps%2C241&sr=8-1-spons&ufe=app_do%3Aamzn1.fos.006c50ae-5d4c-4777-9bc0-4513d670b6bc&psc=1&smid=A29QOP7VOU35ID&spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUExMUM4TUY0MzY3TjRJJmVuY3J5cHRlZElkPUEwNzIwMjM0MThYVlQzQjdNMUZZNSZlbmNyeXB0ZWRBZElkPUEwMTc2MDcyMzI5MUpQWFc2RTVMTSZ3aWRnZXROYW1lPXNwX2F0ZiZhY3Rpb249Y2xpY2tSZWRpcmVjdCZkb05vdExvZ0NsaWNrPXRydWU=">Eaton SPD Type 2</a></p><p>These devices are great, which is why they are now code in all houses. However they still do not react fast enough to protect against a surge from an E1 event. It does <a href="https://youtu.be/bPSTFe1dBAg">sound as though</a> you can use a modified ferrite bead to flatten the initial E1 event down to something one of these surge protectors can handle, but without digging into it, the details were not provided and that could just be snake oil.</p><p><br /></p><h2 style="text-align: left;">Enter the <a href="https://www.empshield.com/">EMP Shield</a></h2><p>EMP shields are advertised as the first whole house surge protectors that react in less than 1 nanosecond and are therefore able to catch an E1 surge without any additional help. Their house unit claims to be able to handle 228 kA, however it <a href="https://youtu.be/B6zsM5U2oNg">sounds as though</a> this is 114 kA per leg, which is probably fine as logically a large surge this device is supposed to handle would probably be split across both incoming power lines anyway.</p><p>The EMP shield is also claimed to be able to handle 90k volts. The amps and volts it can handle are the highest on the market that I have been able to find. However, it is a brand new company started in 2017, so the question is, it is a scam?</p><h4 style="text-align: left;">The Negatives</h4><p>In digging into the EMP shield things start out looking pretty negative for the device. There is not a great deal of data about it being used in the real world, and I have been unable to verify most of their claims. In fact one of the biggest negatives to the EMP Shield is their sales reps.</p><p>In talking with their sales reps the reps come across as being pretty ignorant and only capable of repeating marketing data. They were unwilling or uncapable of sending me a picture of the inside of the device or even better of a third party opening it up and reviewing it. They were even unable to find me third party reviews of their device in action, something any company should have at their finger tips.</p><p>While I did end up finding a couple of decent third party reviews, I was unable to verify the claim that any big energy names or the military were actually purchasing their devices as they claim.</p><p>I did find one very bitter customer who posted on amazon and a couple of other sites claiming that the generator version of the EMP Shield had not protected their generator from a lightening strike despite being installed by a professional electrician. They also claimed the company would not honor the warranty guarantee to fix the damaged equipment.</p><p>I also found a review in which the customer had installed a solar panel version of the EMP Shield and the EMP Shield ended up draining the battery and damaging the battery and potentially the controller. Responses from others on the internet to this review called into question whether the person had installed the correct version of the EMP Shield. Not only are there multiple versions of the shield, there are even multiple versions for different kinds of solar arrays; this is because whole house surge protectors use different versions of metal oxide varistors to shunt energy to ground when it reaches a certain level, and you have to choose the correct one to ensure that the surge protector's resistance is high enough that it does not start shunting at normal usage levels. Based on the lack of customer response to these queries, it seems reasonably possible this was the issue.</p><p>I also asked their rep why their device does not compete better with the Siemens FS140. The rep responded with this:<br /></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;">Siemens has been around for a very long time, we have only been in business a few years. Also, we are in the EMP protection field, they are in the surge protector field. This is different fields so this is no competing.</p></blockquote><p>Given that the EMP Shield is a whole home surge protector advertised to protect against lightening, they are in direct competition with the FS140. And the fact that they are claiming faster reaction times, plus the ability to handle larger loads, and a more diverse line of products, means that they should be beating the FS140 at every corner; except brand name recognition.</p><p>In a Siemens FS140 review the electrician recommends only installing whole home surge protectors with at least a 10 guage wire. He explained that this recommendation is due to larger guage wires being able to handle surges for longer periods before burning up and it is a mark of quality in the device. So I asked the Shield rep about their device, and they responded with:<br /></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;">All of our models have 10 gauge wires except the vehicle model it is 12 gauge wire.</p></blockquote><p>And yet when I read through their <a href="https://www.empshield.com/wp-content/uploads/2022/06/EMPShield-Installation-Instructions-01JUN22.pdf">installation PDF</a> for the house version, I found this:<br /></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p>The EMP Shield (RL – Version) comes with a terminal block 12 AWG COPPER WIRE. tighten to Torque 25 in-lbs. </p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p>The EMPShield (W – version) comes with preconnected 12 AWG COPPER WIRE. Tighten to Torque 25 in-lbs.</p></blockquote><p>Which makes me believe the wires are only 12 guage. I did notice that all their advertising pictures show unmarked wires, however in one video review the wires do have their specifications stamped on them. This could indicate lazy marketing, or a product that was upgraded at a later date.</p><p>Unfortunately after going back and forth with EMP Shield reps multiple times and getting answers that just weren't making sense I asked to speak with someone higher up. They claimed they would have their marketing director reach out to me; however after a month and a couple of follow up emails I have not heard from him at all.</p><h4 style="text-align: left;">The Big Questions</h4><p style="text-align: left;">Did it ever save a house from a lightening strike, and was there third party verification not on their website to support that fact? After watching quite a few videos of people fawning over the device and not knowing anything about it, I finally found <a href="https://youtu.be/OrQrtuWsX-0">what I was looking for</a>. In this video the guy shows how the device shorted the lightening strike and in the process self destructed and even got blown off the wall it was mounted on. Maybe in time we will see more videos like that, but at least we have one, so that is one point in EMP Shields favor.</p><p style="text-align: left;">Can it react in less than 1ns? To backup their claim, EMP Shield has posted <a href="https://www.empshield.com/wp-content/uploads/2019/11/EMP_Shield_Military_Testing_16_March_2019_Public2.pdf">testing results</a> from having their device tested at the Keystone Compliance facility. Since I am not an electrician it took me quite some time to try and figure out what the results were saying. I have two concerns with this document: the first is that there are reaction times listed from .24ns all the way up to 2.08ns. And anything above 1ns would not protect against an E1 event and invalidates their marketing. The second issue is that all the voltages and graphs list the testing power in volts and amps, not in kilovolts and kiloamps. The caveat to this second issue is that the device does protect against lightening, so I must just not understand the graph labeling.</p><p style="text-align: left;">How would it function to protect a car when a car has no mechanism for grounding? This one has been really hard to figure out, and the area where the EMP shield draws the most criticism online from people using common sense and saying it just isn't possible. I watched this <a href="https://www.youtube.com/watch?v=5xeiBMB5DiY">interview</a> with the founder of EMP Shield, and he claimed that all they needed to do was draw the free electrons off the shell of the car itself; and he alludes to them accomplishing that by piping them into the battery. I can get on board with the premise that if you can ground the shell then that will protect what is inside (you'd have to ground the antenna too). But I have a harder time believing that you can just send all that energy straight into the battery and not have the battery blow up. However, by happy accident, I ran across a video that appears to be the emp shield <a href="https://youtu.be/f5wJceJo80o">saving a car</a> from a lightening strike. The video felt authentic, and if it can accomplish this then that certainly makes the EMP claims much more believable.</p><h4 style="text-align: left;">Interesting Tidbits</h4><div>In talking with the EMP Shield rep I got a few pieces of data I am not quite sure what to do with. So i'll list them here and let you make up your mind.</div><div><br /></div><div>I asked why the EMP Shield could handle multiple EMP strikes, but only one lightening strike. The rep claimed it was because an EMP has far less voltage than a lightening strike. A quick search of the Internet does not show a consistent consensus. Estimates for lightening strikes include: 40 kV to 120 kV and 100,000 kV to 1,000,000 kV. And EMP on the other hand is estimated to reach an atmospheric saturation level at 50 kV, with some evidence showing that modern devices might be able to overcome that saturation limit and take it up to 100 kV. So we can assume that the Shield company is assuming an EMP will be limited to the 50 kV estimate.</div><div><br /></div><div>The rep did verify a couple of common sense items. An EMP is a repetitive and long lasting event which is why the devices needs to be able to handle multiple strikes. A lightening strike is a single event, so it is ok for the device to sacrifice itself to handle it. Should an event trip the EMP's breaker(s) then the system will be unprotected until the breaker has been manually reset. In theory any event big enough to trip the breaker should also trip the main breaker which cuts the system off from the grid so it is protected. I have heard some theories that lightening and EMPs can jump small gaps such as those found in breakers, however this is speculative at best.</div><div><br /></div><div>An E1 and an E2 events generally both happen in less than a second, while the E3 event is generally assumed to last a few minutes, although EMP Shield believes it could last as long as several hours. After research, I have no problem believing that 10 guage wires can shunt a large amount of electricity for less than a second without burning up. But minutes or hours of high voltage would easily burn up such small wires. The reps vague and illogical response to this:<br /> </div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: left;">no energy just "sits" in the wires It constantly shunts until all the over voltage is gone.</div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;">an E3 lasts for minutes to hours. It is continuous, and you are correct, without an EMP Shield is does cause problems for wires. that is why the E3 is what will destroy the grid. The E1 and E2 will not. </p></blockquote><p>Constantly "shunting" is a fancy term for energy constantly flowing through the wires to ground, which sounds like the very definition of "sitting" in the wires. Especially if we are talking about high voltages over a several hour period. The rep claims that an EMP will not trip the breakers. So the rep admits that an hours long event would destroy the wiring in the grid, yet somehow the tiny wires in the Shield can handle it for an extended period because they are grounded; I am having trouble accepting this claim.</p><p>There is also the odd add on of the claim that the E1 and E2 events will not cause a problem for the grid. My guess is since these are shorter wave events and can more easily bind to smaller wires the Rep was insinuating that the E3 event would be the one more likely to bind to the long electrical wires. It felt poorly communicated to me.</p><p>When installing the EMP shield in a house, it is important that the 20 amp breaker(s) cover both rails, a double pole breaker does this naturally. It is also best to have the breaker be as close to the main wires coming into the house as possible to create the fastest path to ground for the extra energy.</p><div></div><h4 style="text-align: left;">Conclusion</h4><p>After waiting a month with several follow-up emails for EMP Shield to answer these troubling discrepancies in their product, advertising, and commutation, I am left to assume that I will not be hearing back or get these questions answered.</p><p>Ultimately I do not feel solid in the idea that the EMP shield will do what it claims. However, with the two videos I found, and the conceptual research I have done I'd say there is a small possibility that the EMP Shield will function as advertised; or at least function as a basic whole home surge protector. Given how sketchy things are around this company I will not personally be purchasing one at this time.</p><p><b>EDIT ( 3/13/2024 )</b></p><p>We now finally have <a href="https://www.youtube.com/watch?v=5zW4UMLEciU">a YouTube video</a> of an independent party who knows what they are doing actually testing the EMP Shield device. Despite all my misgivings prior, this video pushes it over the edge for me, and I now intend to get one as soon as finances allow.</p><p>In case the video does not exist by the time this post is being read, in summary a guy sets up a lab with the equivalent of a giant tesla coil type enclosed system on one end, a bunch of sensitive house hold electronics in the middle of the circuit, and the emp shield house device at the end of the circuit. The whole system is powered by 120VDC.</p><p>The guy was doing his best to break the emp shield because he did not believe their claims. He had an 8" spark gap setup to show the level of energy he was injecting into the system. The capacitors on the firing end would charge using the 120VDC then release creating a giant surge crossing that spark gap and hitting the entire system.</p><p>As part of the demonstration he showed all the devices working, and talked about how the test was affecting the rest of his house despite his house being disconnected from the testing room aside from the 120V source current.</p><p>After hitting the circuit over 100 times the emp shield device was still working and no devices appeared damaged in anyway. I believe he mentioned a couple of them might have rebooted, that was the extent of the damage.</p><p>He then showed a brief clip of him hitting the circuit with the emp shield device disconnected and every device on the circuit was badly destroyed. He also described how the destruction was so bad that the majority of his closeup video footage was lost because the camera filming it had been destroyed as well.</p><p>So not only does the emp shield protect the circuit it is on, it prevents a lot of residual radiation from entering the air around the circuit. This was a very well done test by a third party. The test was being discussed and reviewed by a panel of 4 guys with various expertise's in the field, and electrician, a bomb tech, and a prepper were among them.</p>Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-67603008136316137582022-02-18T09:32:00.001-08:002022-02-18T09:32:12.769-08:00Installing VeraCrypt on Raspberry Pi4<br /><div style="text-align: center;"><b>Raspberry Pi 4 Project</b></div><div style="text-align: center;"><br /></div><div><div>I have been interested in small computing devices for awhile starting with various Arduino projects. Recently I took on a new Raspberry project where I needed it to be able to read encrypted USB drives. I ran into several issues along the way that I intend to document here to help others attempting similar things.<div><br /></div><div>When working with a PI I generally use the Raspbian OS, it has been the one I have had the most success with. For this project I tried both the Lite and the Full versions of the OS, and while they were both successful, the full version had one feature that caused me to go with it over Lite.</div><div><br /></div><div style="text-align: center;"><b>USB exFat Issue</b></div><div><br /></div><div>The USB drives I was using were 64GB, which means they were too big for the older FAT32 format and were formatted exFat. The full version of Raspbian auto mounted and read them perfectly, while the Lite version could not and needed the <a href="https://pimylifeup.com/raspberry-pi-exfat/">exFAT libraries installed</a> using the following two commands:</div><div><br /><div><blockquote>sudo apt-get install exfat-fuse<br />sudo apt-get install exfat-utils</blockquote><br />I didn't have any problem installing the libraries, but after install the drives would still not auto mount as the documentation indicated they should. They did work correctly after performing a manual mount using the following command:<br /><br /><blockquote>sudo mount -t exfat /dev/sda1 /mnt/usb1</blockquote></div><div><br /></div><div>Part of this project was having an environment that wouldn't take me too long or too much work to re-create in the future if my micro SD card died. So rather than continue to fight with why the USBs were not auto mounting, I ditched my preferred Lite version of Raspbian, and went with the Full version.</div><div><br /></div><div style="text-align: center;"><b>VeraCrypt</b></div><div><br /></div><div>Now that my external USB drives were working correctly "out of the box", all I really needed to do was install the VeraCrypt software. I use VeraCrypt because it is both free and one of the simplest and most portable encryption options available.</div><div><br /></div><div>The first place I looked for a package was the <a href="https://veracrypt.fr/en/Downloads.html">recent downloads</a>. I grabbed the armhf version which is advertised as working on the 32bit Raspbian OS. After installing and attempting to run it I was presented with the following errors:</div><div><br /></div><div><blockquote>veracrypt: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by veracrypt) veracrypt: /usr/lib/arm-linux-gnueabihf/libwx_gtk3u_core-3.0.so.0: version `WXU_3.0.5' not found (required by veracrypt)</blockquote></div><div><br /></div><div>I founded <a href="https://raspberrypi.stackexchange.com/questions/135554/veracrypt-on-rpi-4-buster-unable-to-find-libraries">one post</a> that indicated the errors meant I had the wrong version of the libraries installed. It took me awhile to figure out the actual names of the libraries the dependencies were referring to, however <a href="https://codeyarns.com/tech/2019-05-17-how-to-install-veracrypt-on-raspbian.html">Code Yarns</a> finally solved that problem for me telling me to run this command.</div><div><br /></div><div><blockquote>sudo apt install libfuse-dev libwxbase3.0-dev</blockquote></div><div><br /></div><div>Unfortunately I was still getting the same error. Considering Ashwin from Code Yards was successful in getting it to work, and remembering the errors were supposed to be version issues, not just if the libraries were installed at all, I decided to see if the version of VeraCrypt Ashwin used would work.</div><div><br /></div><div>But of course I didn't want to go all the way back to his version, so I started walking my way back trying each version to see if one would work. Not a particularly fun task as VeraCrypt is not compiled for Raspian with each iteration, you have to go into each folder and see if you can find an armhf version compiled. Several people solved this issue by simply compiling the latest version of VeraCrypt for themselves; but again, I was trying to go for the most easily reproducible build.</div><div><br /></div><div>I finally found a working version in <a href="https://sourceforge.net/projects/veracrypt/files/VeraCrypt%201.24-Update7/">VeraCrypt 1.24 Update 7</a>. My guess is that if I had looked for newer versions of libfuse and libwxbase I might have been able to use a newer version of VeraCrypt as well.<br /></div></div></div></div><div><br /></div><div>After that it was all down to testing the install and making sure encrypted files could be created and opened. I had no more issues afterwards, the Raspberry Pi device is really becoming quite the powerful tool these days.</div>Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-7206512302169473492022-02-15T08:02:00.007-08:002022-02-15T08:06:11.385-08:00SQL Server fast failover in High Availability Groups<p> I recently had the somewhat dubious honor of helping migrate a company from a physical data center to the Azure cloud. I say dubious because for as many positives as there are about Azure, there are at least the same number of negatives. The only feature Azure has that makes it worth migrating, in my opinion, for most smaller companies is the ability to work and see the environment the same way a data center would see it; so much more visibility, not good visibility, just visibility.</p><p>However, my feelings about Azure are not the point of this post. As a part of this migration we went from a traditional failover SQL cluster to an Always on High Availability group; primarily because the dba doing the migration could not figure out how to do a failover cluster in Azure.</p><p>In my short exposure to the high availability option it is both good and bad. The automated failover in SQL server is faster than it was using the old method, however the visibility into the individual server databases is worse.</p><p>The DBA handling the migration was unable to figure out how to manually fail back the system in a reasonable time frame. He was using a SQL script that failed back one availability group at a time. This approach would leave our system down for up to a minute, a time frame that would increase as we added more databases to the system, and this was a big problem.</p><p>To solve this problem I decided to go with a multi-threaded approach so I could fail back every availability group simultaneously. Unfortunately SQL does not seem to have any form of reasonable multi-threaded capability available for a script to use. So I turned to scripting languages.</p><p>While this script could be written in most scripting languages, I chose the older VBscript this time. The general idea is that the script loops through all found availability groups, and spins up an instance of itself in a separate thread to start a failover for each one.</p><p></p><br /><div></div><blockquote><div>Const sqlUser = "<username>" ' add your server admin login username here</div><div>Const sqlPass = "<password>" ' add the password for the username here</div><div>Const Server = "<ip address>" ' enter the ip address of the DESTINATION server to fail to here</div><div><br /></div><div>Const adOpenStatic = 3</div><div>Const adLockOptimistic = 3</div><div><br /></div><div>'</div><div>' Here is where we fail over the current AG</div><div>'</div><div>If (WScript.Arguments.Count > 0) Then</div><div> 'Wscript.Echo WScript.Arguments.Item(0)</div><div><span style="white-space: pre;"> </span></div><div><span style="white-space: pre;"> </span>Set failConnection = CreateObject("ADODB.Connection")</div><div> failConnection.Open _</div><div> "Provider=SQLOLEDB;Data Source=" + Server + ";" & _</div><div> "Trusted_Connection=No;Initial Catalog=master;" & _</div><div> "User ID=" + sqlUser + ";Password=" + sqlPass + ";"</div><div><span style="white-space: pre;"> </span></div><div><span style="white-space: pre;"> </span>failConnection.Execute "ALTER AVAILABILITY GROUP " + WScript.Arguments.Item(0) + " FAILOVER;"</div><div><br /></div><div><span style="white-space: pre;"> </span>failConnection.Close</div><div><span style="white-space: pre;"> </span>Set failConnection = Nothing</div><div><span style="white-space: pre;"> </span></div><div> WScript.Quit 0</div><div>End If</div><div><br /></div><div><br /></div><div><br /></div><div>'</div><div>' Below is where we get the list of AGs to fail over</div><div>'</div><div><br /></div><div><br /></div><div>Dim FSO</div><div>Set FSO = CreateObject("Scripting.FileSystemObject")</div><div>GetCurrentFolder = FSO.GetAbsolutePathName(".")</div><div><br /></div><div>Set objConnection = CreateObject("ADODB.Connection")</div><div>Set WshShell = WScript.CreateObject("WScript.Shell")</div><div><br /></div><div><br /></div><div>objConnection.Open _</div><div> "Provider=SQLOLEDB;Data Source=" + Server + ";" & _</div><div> "Trusted_Connection=No;Initial Catalog=master;" & _</div><div> "User ID=" + sqlUser + ";Password=" + sqlPass + ";"</div><div><br /></div><div>Set objRecordSet = objConnection.Execute("SELECT name FROM master.sys.availability_groups")</div><div><br /></div><div>objRecordSet.MoveFirst</div><div><br /></div><div>Do Until objRecordSet.EOF</div><div> 'Wscript.Echo objRecordSet.Fields("name")</div><div><span style="white-space: pre;"> </span></div><div><span style="white-space: pre;"> </span>WshShell.Run Wscript.ScriptFullName + " " + objRecordSet.Fields("name")</div><div><span style="white-space: pre;"> </span>objRecordSet.MoveNext</div><div>Loop</div><div><br /></div><div>objRecordSet.Close</div><div>objConnection.Close</div><div>Set objConnection = Nothing</div><div><br /></div><div>Wscript.Echo "All failover commands fired"</div></blockquote><p><br /></p><p>With this approach the entire fail back is reduced to a couple of seconds, and the time does not increase noticeably as additional groups get added in the future.</p><div></div> <p></p>Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-21172588176239440942021-12-10T13:33:00.004-08:002021-12-10T13:35:04.220-08:00Converting Google Spreadsheet App Script to Libre Office Macro<p> I recently had a task to convert a small macro that did some data copying in a Google Spreadsheet, into an equivalent version in Libre Office.</p><p>Google has a very JavaScript like feel to it's scripting language, while Libre/Open Office uses either Python or Basic as their languages. Basic seemed like a closer format to what I was coming from, and I've had some experience with VBA in the past, so I chose to go with that over the more powerful Python alternative.</p><p>My primary purpose in posting this is that there seems to be a lack of documentation around the Libre Office Basic language, which is understandable given the support of Python. But in case someone else really feels like using Basic for a small project like I did, hopefully this will help them find some of the functions they need.</p><p>Here is the Google macro I was trying to convert</p><p></p><blockquote><p>function fCopyData() {<br /> // get all the required sheets<br /> var shtSum = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Source");<br /> var shtFut = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Dest");</p><p> // get the values we will be working with<br /> var vals = shtSum.getRange("A7:D45");<br /> var vIn = shtSum.getRange("I2:K10");</p><p> // if we don't have enough rows, add more<br /> if((shtTran.getMaxRows()-shtTran.getLastRow()) < vals.getNumRows()) {<br /> shtTran.insertRows(shtTran.getMaxRows(),vals.getNumRows()-(shtTran.getMaxRows()-shtTran.getLastRow())+2);<br /> }</p><p> if((shtFut.getMaxRows()-shtFut.getLastRow()) < vals.getNumRows()) {<br /> shtFut.insertRows(shtFut.getMaxRows(),vals.getNumRows()-(shtFut.getMaxRows()-shtFut.getLastRow())+2);<br /> }</p><p> // populate dest data<br /> var today = new Date();<br /> today.setDate(today.getDate()+7);</p><p> var x = 1;<br /> while(x <= vals.getNumRows()) {<br /> if(vals.getCell(x,1).getValue()=="ENV") {<br /> var lastRow = shtFut.getLastRow()+1;<br /> shtFut.getRange(lastRow,1).setValue(vals.getCell(x,4).getValue()*-1); // col 1<br /> shtFut.getRange(lastRow,2).setValue((today.getMonth()+1)+'/1/'+today.getYear()); // date<br /> shtFut.getRange(lastRow,3).setValue('=Year(B'+lastRow+')'); // year<br /> shtFut.getRange(lastRow,4).setValue('=Month(B'+lastRow+')'); // month<br /> shtFut.getRange(lastRow,5).setValue(vals.getCell(x,2).getValue()); // category<br /> shtFut.getRange(lastRow,6).setValue(vals.getCell(x,3).getValue()); // description<br /> }<br /> x++;<br /> }</p><p> Browser.msgBox("Finished!");<br />}</p></blockquote><p></p><p><br /></p><p>Google has that special feature where you actually have to add new rows when you want to use them in their spreadsheets. Moving to a desktop app no longer had that restriction, so part of the code could just be removed which was nice.</p><p>However, figuring out all the correction function names to take the place of the Google equivalents proved to be very difficult.</p><p>So here is the equivalent in Libre/Open Office Basic</p><p></p><blockquote><p>REM ***** BASIC *****</p><p>Sub CopyData<br /> Dim Document As Object<br /> Dim Sheets As Object<br /> Document = ThisComponent 'assigns the current document to the variable document<br /> Sheets = Document.Sheets 'get the container of all Sheets</p><p> ' get all the required sheets<br /> Dim shtTran As Object<br /> Dim shtSum As Object<br /> Dim shtFut As Object<br /> shtSum = Sheets.getByName("Source")<br /> shtFut = Sheets.getByName("Dest")</p><p> ' get the values we will be working with<br /> Dim vals As Object<br /> Dim vIn As Object<br /> vals = shtSum.getCellRangebyName("A7:D45")<br /> vIn = shtSum.getCellRangebyName("I2:K10")</p><p> ' populate dest data<br /> Dim thisYear As String<br /> Dim thisMonth As String<br /> Dim lastRow As Integer<br /> thisYear = Format(Now() + 7, "YYYY")<br /> thisMonth = Format(Now() + 7, "MM")</p><p> oCursor= shtFut.createCursor<br /> oCursor.gotoEndOfUsedArea(False)<br /> lastRow = oCursor.RangeAddress.EndRow</p><p> Dim Cell as Object<br /> Dim x As Integer<br /> For x = 0 To (vals.Rows.getCount() - 1)<br /> If vals.getCellByPosition(0,x).String = "ENV" Then<br /> lastRow = lastRow+1</p><p> ' col 1<br /> Cell = shtFut.GetCellByPosition(0,lastRow)<br /> Cell.Value = vals.getCellByPosition(3,x).VALUE*-1<br /> Cell.NumberFormat = 125 ' secret code for Currency Format</p><p> ' date<br /> Cell = shtFut.GetCellByPosition(1,lastRow)<br /> Cell.Value = DateValue(thisMonth & "/1/" & thisYear)<br /> Cell.NumberFormat = 36 ' secret code for Currency Format</p><p> ' year<br /> Cell = shtFut.GetCellByPosition(2,lastRow)<br /> Cell.Formula = "=YEAR(B"+(lastRow+1)+")"<br /> Cell.NumberFormat = 0 ' Numeric Format</p><p> ' month<br /> Cell = shtFut.GetCellByPosition(3,lastRow)<br /> Cell.Formula = "=Month(B"+(lastRow+1)+")"<br /> Cell.NumberFormat = 0 ' Numeric Format</p><p> ' category<br /> Cell = shtFut.GetCellByPosition(4,lastRow)<br /> Cell.String = vals.getCellByPosition(1,x).String</p><p> ' description<br /> Cell = shtFut.GetCellByPosition(5,lastRow)<br /> Cell.String = vals.getCellByPosition(2,x).String<br /> End If<br /> Next x</p><p> msgbox "Finished!"<br />End Sub</p></blockquote><p></p><p><br /></p><p><br /></p>Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-55170648406077334622020-08-10T22:41:00.002-07:002020-08-10T22:43:01.466-07:00Milestone Screen Recorder changes Default username<p> I've been using the Milestone XProtect VMS system for several years now, and it has had various issues, many of which they have solved over time.</p><p>One of their addons, called the Screen Recorder, is a particularly handy piece of software, but has caused an outsized number of headaches given the simplicity of it.</p><p>It allows you to record a computer screen pretending as though the computer screen were a camera so all the recording footage is stored along side of all your other video footage in your VMS system.</p><p>There are multiple very helpful uses for it, one of the best is in recording customer facing demo systems to make sure people aren't doing things they should not be.</p><p>For many years it had a very odd bug in it where if the computer was turned off for too long (days), then the recording server would eventually stop trying to contact it, so when the computer was turned back on, the recording process would not be re-initiated. However, a recent fix seems to have solved that bug.</p><p>A month ago I upgraded my demo system to the 2020R2 version of the recording server, and for the last couple of weeks I have been struggling trying to get a couple of screen recorders to come back online. Traditionally in the past the big "gotcha" with the screen recorder has been that the username is hard coded to "Milestone" with a capital M. If you don't know that then you will be trying all sorts of default usernames trying to figure out how to make it work; and of course it is not configurable on the client side, it was also not super obvious in the milestone documentation. You had to go through their support sites to figure it out in most cases.</p><p>Their documentation has gotten significantly better the last couple of years, and I finally decided to re-read the screen recorder install documentation to see what I might have missed since all the old tricks I had learned over the years were not working. Low and behold, it seems as though they change the hard coded username from "Milestone" with a capital M, to "videoos", as <a href="https://doc.milestonesys.com/mc/pdf/2020r2/en-US/MilestoneXProtectScreenRecorder_AdministratorManual_en-US.pdf">described here</a>.</p><p>I had to laugh when I found this change, things worked so smoothly after figuring that out. But the change is amusing because I can think of no good reason for it. It is going to irritate all their clients who learned the old way, it will not increase security because it is still a hard coded username like before, and you still can't alter it on the client machine. I am happy they at least thought to document the unannounced change though as it has not yet made it to any support forum I have found.</p>Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-58667709177989202212020-03-17T19:33:00.001-07:002020-03-17T19:33:39.026-07:00Syncing a similar code base between multiple client instancesFor years one of the companies I worked for has juggled multiple copies of a very similar code base. The majority of the code was (or could be) identical, but there was just enough of a difference that a single code base could not be used. An additional complicating factor was that the company was known by its clients for the ability to quickly turn around feature change requests; which meant that when one client wanted a change, there was not time to test its impact for all clients before rolling it out.<br />
<br />
As you can imagine, this was a very difficult process to manage, and very time consuming re-creating identical features when other clients decided they wanted something that had been developed.<br />
<br />
Over the years multiple attempts have been made to solve this issue.<br />
- Branching was scrapped because changes can really only be pushed from the base branch to the child/client branches and there was not generally time to test all feature updates when a client wanted just one single change pushed into their code base.<br />
- A shared service architecture was scrapped because versioning quickly became unwieldy between the clients using it, and the shared services started to become fractured. They also suffered from the same inability to easily test and regression test all combinations of the endpoints. Also a shared database became a security concern.<br />
- A shared dll was scrapped for similar reasons when one client updated the dll, and the other clients were forced to take all the updates on their next modification.<br />
- Splitting the code into multiple projects by major feature area. This allowed for smaller pushes when changes were made, and attempts could be made to keep the most similar projects in sync, but it was still unwieldy.<br />
<br />
As the features became more numerous regression testing became a big issue. So a fairly comprehensive automated unit test and UI testing system was developed. This significantly reduced the danger of moving features between the client code bases, but it did nothing to reduce the time involved.<br />
<br />
A lot of <a href="https://stackoverflow.com/questions/1075708/diff-utility-works-for-2-files-how-to-compare-more-than-2-files-at-a-time">posts</a> were reviewed, and a lot of tools tried in an effort to figure out how to have a human easily view and push changes around between all the code bases.<br />
<br />
- <a href="https://www.2brightsparks.com/syncback/sbpro.html">SyncBackPro</a> (great sync tool, but no human review during the process)<br />
- <a href="https://winmerge.org/">Winmerge</a><br />
- <a href="https://www.vim.org/">Vim</a><br />
- <a href="http://diffuse.sourceforge.net/">Diffuse</a> (amazing tool, but crashes on windows with three or more files open)<br />
- <a href="https://www.devart.com/codecompare/">Code Compare</a> (best code compare tool found, but only supported up to three files)<br />
<br />
Over time Code Compare became the hands down favorite in the company for comparing code files, it was a smoother compare process for code files than any other tool tried; and made pushing changes around much easier.<br />
<br />
However, even though it was smoother, it was still a massively huge process; and getting bigger as more and more features were added. <a href="https://softwareengineering.stackexchange.com/questions/60393/how-to-maintain-different-customized-versions-of-the-same-software-for-multiple">Others</a> have had similar problems, but no one had any amazing and workable general solutions; although many people had tried playing with various git branching type features. One company even developed a<a href="https://github.com/teambit/bit"> piece of software</a> attempting to tackle this problem, however it appears to be primarily for UI development rather than backend code.<br />
<br />
In this ever evolving situation the next attempt build on the popular Code Compare. A custom powershell script was developed that mapped code bases to either other, or back to a master allowing the code to be instantly compared using Code Compare between a "master" instance of the code so the developer could push new features back to master, and pull down any desired feature differences.<br />
<br />
The custom script:<br />
- installs/removes itself in the right click menus for files and folders<br />
- includes directions for installing itself in those menus inside of Visual Studio<br />
- requires the code bases being compared to have identical directory and file structures<br />
- requires that you go through and modify the $parentProjects variable to map your project folders<br />
- uses the $parentProjects mapping to detect the incoming file/folder path, and open a comparison with Code Compare to the file/folder of the corresponding mapped path<br />
<blockquote class="tr_bq">
</blockquote>
Here is the Custom Power Shell Script<br /><blockquote class="tr_bq">
<br />#<br /># VISUAL STUDIO INSTALLATION INSTRUCTIONS:<br /># Menu: Tools / External Tools<br /># - Click Add<br /># - Title: Mas Code Compare<br /># - Command: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe<br /># - Arguments: -File "D:\Documents\scripts\CompareProjects.ps1" "$(ItemPath)"<br /># - memorize the index number of the item you just created (how far down in the list it is)<br /># Menu: Tools / Customize<br /># - Click the Commands tab<br /># - choose Context menu: Project and Solution Context Menus | Item<br /># - - Add Command...<br /># - - Choose the category Tools<br /># - - Select External Command <index external="" from="" number="" tool=""><br /># - choose Context menu: Project and Solution Context Menus | Folder<br /># - - Add Command...<br /># - - Choose the category Tools<br /># - - Select External Command <index external="" from="" number="" tool=""><br />#<br /><br /><br /><br /><br /><br />if ($args[0] -eq $null) {<br /> msg *, "Run with the following parameters: -install, -remove, 'PathOfFileOrFolder'"<br /> return<br />}<br /><br />#<br /># Install or Remove the windows context menu item<br />#<br />if ($args[0] -eq "-install" -or $args[0] -eq "-remove")<br />{<br /> # AllFilesystemObjects is the key folder here, it specifies that the "shell" sub folder will be applied<br /> # to all file system objects.<br /> # the "shell" sub folder indicates that we are dealing with the right click context menu<br /> # and the final folder name becomes the name of the menu item itself<br /> $registryPath = "HKCR:\AllFilesystemObjects\shell\MasCodeCompare"<br /> $regAutoPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Auto"<br /> $regMasCodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.MasterCode"<br /> $regCli1CodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Client1Code"<br /> $regCli2CodePath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\CommandStore\shell\MasCodeCompare.Client2Code"<br /> New-PSDrive -Name HKCR -PSProvider Registry -Root HKEY_CLASSES_ROOT<br /> New-PSDrive -Name HKLM -PSProvider Registry -Root HKEY_LOCAL_MACHINE<br /><br /> if ($args[0] -eq "-install")<br /> {<br /> # attempts to get command windows to not show with (/Q and -windowstyle hidden) don't work<br /> $Name = "(Default)"<br /> $value = "CMD.EXE /Q /C Powershell.exe -windowstyle hidden -File "+$PSScriptRoot.replace("\", "\\")+"\\CompareProjects.ps1 %1"<br /><br /> IF(!(Test-Path $registryPath))<br /> {<br /> New-Item -Path $registryPath -Force | Out-Null<br /> New-Item -Path ($registryPath + "\command") -Force | Out-Null<br /> New-Item -Path $regAutoPath -Force | Out-Null<br /> New-Item -Path ($regAutoPath + "\command") -Force | Out-Null<br /> New-Item -Path $regMasCodePath -Force | Out-Null<br /> New-Item -Path ($regMasCodePath + "\command") -Force | Out-Null<br /> New-Item -Path $regCli1CodePath -Force | Out-Null<br /> New-Item -Path ($regCli1CodePath + "\command") -Force | Out-Null<br /> New-Item -Path $regCli2CodePath -Force | Out-Null<br /> New-Item -Path ($regCli2CodePath + "\command") -Force | Out-Null<br /> }<br /><br /> New-ItemProperty -Path ($registryPath + "\command") -Name $name -Value $value -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path $registryPath -Name "MUIVerb" -Value "Mas Code Compare" -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path $registryPath -Name "SubCommands" -Value "MasCodeCompare.Auto;MasCodeCompare.MasterCode;MasCodeCompare.Client1Code;MasCodeCompare.Client2Code;" -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path $regAutoPath -Name "MUIVerb" -Value "Auto choose" -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path ($regAutoPath + "\command") -Name $name -Value $value -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path $regMasCodePath -Name "MUIVerb" -Value "Master Code" -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path ($regMasCodePath + "\command") -Name $name -Value ($value + " Company\MasterCode") -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path $regCli1CodePath -Name "MUIVerb" -Value "Client 1" -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path ($regCli1CodePath + "\command") -Name $name -Value ($value + " Client\Client1Code") -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path $regCli2CodePath -Name "MUIVerb" -Value "Client 2" -PropertyType String -Force | Out-Null<br /> New-ItemProperty -Path ($regCli2CodePath + "\command") -Name $name -Value ($value + " Client\Client2Code") -PropertyType String -Force | Out-Null<br /> return<br /> }<br /> if ($args[0] -eq "-remove")<br /> {<br /> if (test-path $registryPath) { remove-item $registryPath -Recurse }<br /> if (test-path $regAutoPath) { remove-item $regAutoPath -Recurse }<br /> if (test-path $regMasCodePath) { remove-item $regMasCodePath -Recurse }<br /> if (test-path $regCli1CodePath) { remove-item $regCli1CodePath -Recurse }<br /> if (test-path $regCli2CodePath) { remove-item $regCli2CodePath -Recurse }<br /> return<br /> }<br />}<br /><br />#<br /># If we have gotten to here, then we are probably trying to do a compare<br />#<br />$itemPath = $args[0]<br />$compareProject = $args[1]<br /><br /><br /><br />$parentProjects = @{<br /> "Client\Client2Code" = "Company\MasterCode";<br /> "Client\Client1Code" = "Company\MasterCode";<br /> "Company\MasterCode" = "Client\Client1Code"<br />}<br />$parentProject = $null<br /><br /># find mapping that applies<br />foreach ($proj in $parentProjects.GetEnumerator())<br />{<br /> if ($itemPath -like "*" + $proj.Name + "*") {<br /> $parentProject = $proj<br /> break<br /> }<br />}<br /><br /># if no parent found, alert user<br />if ($parentProject -eq $null)<br />{<br /> Msg * "Invalid project selection"<br /> return<br />}<br /><br /># if a parent wasn't requested, then find parent from mapping<br />if ($compareProject -eq $null)<br />{<br /> $compareProject = $parentProject.Value<br />}<br /><br />$parentPath = $itemPath.replace($parentProject.Name, $compareProject)<br /><br />& "C:\\Program Files\\Devart\\Code Compare\\CodeCompare.exe" "/environment=auto" "$parentPath" "$itemPath"</index></index></blockquote>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-87943351289082409082020-02-24T18:54:00.003-08:002020-02-24T18:54:46.975-08:00Free 000WebHosting GotchaI run a small family genealogy website for my immediate relatives. It is not large, and generates very little traffic, so I have always tried to host it somewhere for free. Most of it's life it lived on Google's free pages. However, with Google's recent upgrade to their free web pages system, that system lost so much functionality that I was forced to look somewhere else for hosting.<div>
<br /></div>
<div>
After comparing multiple offerings I settled on <a href="https://www.000webhost.com/">000webhost.com</a>. They had site limiters and other restrictions on their free offering, but again, all I need was a basic place for my personal family to be able to read a few stories. They also allowed WordPress hosting for free, and I am pretty comfortable with WordPress, so it is my CMS of choice.</div>
<div>
<br /></div>
<div>
Everything went very well for multiple months. We (my family) all worked hard and got the content manually copied from Google's old static page system into the new WordPress system hosted by 000webhost. After the initial copy several months were spent adding new data, and fixing some of the formatting issues that occurred during migration.</div>
<div>
<br /></div>
<div>
During this time period I was creating backups once every couple of months, which is the only thing that ended up saving us.</div>
<div>
<br /></div>
<div>
One day I got an email from a family member saying that the site was telling her it had been deleted. I went to login and sure enough, my cPanel login was gone, the entire website was just gone.</div>
<div>
<br /></div>
<div>
I did a little research in the 000webhost KB system, and found an article that said because it was free hosting, if your site ever got deleted, too bad so sad, it was just gone. I could understand this, it was a free site, that policy made sense; I just didn't know why my site had been deleted.</div>
<div>
<br /></div>
<div>
After contacting 000webhost to see what had happened, I received a very polite email back saying that because I had not logged into the site in a long time, they had deleted it. I was a bit taken aback, we had been actively working on the site every week for months. With a bit more clarification I learned that in order for activity on the site to count, it had to be a login specifically to the cPanel system.</div>
<div>
<br /></div>
<div>
At this point I was finally irritated. It was a free hosting service so any policy they wanted to put in place was fine with me, as long as they were up front about it. And that was my problem, many other sites had been up front about having a policy like that, but 000webhost had not. It might be buried somewhere on their site, but at the time I signed up with them there was no warning that this would occur.</div>
<div>
<br /></div>
<div>
This is where the politeness ended from 00webhost, after expressing my displeasure that my site had been deleted with no warning, and no initial knowledge on my part, the response from them said I should have gotten a warning email, but it didn't always work; and they seemed rather excited to see me go.</div>
<div>
<br /></div>
<div>
Needless to say, I took my most recent backup and went to find another host. There's a decent chance we will start paying for a host eventually as the site grows, but I will never give 000webhost or their parent company <a href="https://www.hostinger.com/">hostinger.com</a> my business after an experience like this.</div>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-64952221717329484402019-11-13T13:18:00.000-08:002019-11-13T13:18:32.126-08:00Accord.net Machine LearningMachine learning has been out of reach of the common developer for much of its early life. While R has come along and stood out as "the" statistics language, it does not easily plug into the more mainstream languages. Fortunately for us, in the last few years a library called <a href="http://accord-framework.net/">accord-framework.net</a> has grown up to fill this gap.<br />
<br />
This framework is written in C# and allows the average .net developer access to a large number of machine intelligence algorithms that require very little statistical knowledge to actually use. I specify the statistical qualifier because it does take a decent amount of basic c# experience to overcome some of the rough edges in validation the library still has.<br />
<br />
The website also includes an impressive amount of documentation with code examples, unfortunately many of these examples are aging and have broken pieces in them due to changes in the software, however they are usually enough to get someone up and going after a little playing around.<br />
<br />
<b>Concepts and Implementation</b><br />
<br />
The concept is fairly simple. All the machine learning algorithms in this library take a two dimensional array of numbers (int or double), along with a single dimensional array of values with the correct outputs. The algorithm then trains itself on these numbers.<br />
<br />
After training you send it another set of numbers in the same format, and this time it will give you back what it thinks the outputs will be.<br />
<br />
Your process will look something like this:<br />
- Load datatable with data<br />
- Codify the datatable into integer values using the Codification library<br />
- Extract a two dimensional integer array using a combination of the datatable and the newly created code mapping object for all the columns you want to use as inputs.<br />
- Extract a single dimensional integer array just like you did the two dimensional one, only this time for the single column that holds the values you are trying to guess. It is important that the order of the values in this single column cause them to correspond with the correct input column values by index location. If all the columns are being extracted from the same datatable then this should happen naturally.<br />
- You will pass these two arrays into the desired algorithm to train it, some algorithms will require multiple training cycles to tune them.<br />
- Once you have a trained algorithm object you can then pass it another two dimensional array of integers. This time the values will be used to guess a single dimensional array of output integer values. One gotcha here is that many of the algorithms can't handle input values they have not been trained on, so you can't throw just anything at it.<br />
<br />
Because it works only with integers, any string values you want to use as input or output must first be converted to integers with no gaps between the number values. You can do this on your own, but for convenience they have provide a special Code library which handles converting standard tables of data into encoded integers and back.<br />
<br />
<b>Gotchas</b><br />
<br />
There are multiple bugs and missing features in this Code library, which is one of the biggest challenges that has to be overcome when working with these algorithms. However, despite these issues I have still chosen to use the conversion library.<br />
<br />
I have discovered that it's biggest shortcoming is that it is not capable of handling NULL values in the data. So first you have to loop through every single value in your datatable and remove all NULL values. From a speed perspective, this flaw alone probably means it would make for faster code to roll your own; but for the majority of developers out there, it is likely not worth the additional time to do that.<br />
<br />
I have read that there is a default value setting inside the library at a per column level that allows the library to deal with NULLs. For some reason that default either does not work, or is not initialized on its own.<br />
<br />
The next annoying issue this Codification library has is really more of a versioning problem. It looks like over time, rather than fixing particular issues, new methods get created to handle the new cases. So you end up with multiple methods that run off different logic when encoding values. Specifically, there seems to be a big difference in Codification between the Transform method and the Apply method. Transform seems to attempt to specifically encode all columns requested, kind of like the explicit class creation overload that accepts a list of columns. Apply on the other hand logically processes a datatable detecting which columns need to be converted and which do not.<br />
<br />
<b>Thoughts and Concepts</b><br />
<br />
Most of my work in this area has been with the various <a href="https://github.com/accord-net/framework/wiki/Classification">Classification algorithms</a> in the accord.net library. These seem to have the limitation of not being able to accept a Continuous (un-encoded int) value type as the output column it is supposed to be guessing. The solution to this particular issue is probably to switch to using a Regression algorithm.<br />
<br />
Something else that might not immediately come to the mind of the average developer is that the output is not going to have any human readable meaning. Because the system works exclusively with integer arrays, the output will just be an array of numbers. These output numbers must then be passed through the Codification library a second time, this time in reverse, to get back to the human readable version of them.<br />
<br />
<br />
<br />Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-2686592077739407832019-06-19T20:12:00.001-07:002019-06-19T20:12:55.278-07:00Microsoft Owin Http append header bugI've been dealing with an exception for most of this year that has really been annoying.<br />
<br />
This bug is specific to .net Owin code, possibly even the CookieAuthentication method when running under IIS in an integrated pipeline. While this sounds very specific, I believe it is actually one of the more common setups for .net and IIS.<br />
<br />
After the first couple of months I looked into it and found that <a href="https://github.com/aspnet/AspNetKatana/issues/74">others were having</a> the same issue. Unfortunately, Microsoft already knew about it and responded that the fix for it was such a low priority that they were not going to bother with it any time soon.<br />
<br />
After many months of dealing with this annoying bug I finally found time to spend several days digging into Microsoft's code to see what was causing it.<br />
<br />
As the Microsoft guy <a href="https://github.com/aspnet/AspNetKatana/issues/74#issuecomment-362428094">had indicated</a>, it was the <a href="https://support.microsoft.com/en-us/help/2901671/child-requests-generate-duplicate-asp-net-events-in-iis">parent/child request</a> scenario in IIS that was triggering the issue. When I first read his response I did not really understand what he meant by it, or how it was triggering the errors.<br />
<br />
<b>Recreating the error</b><br />
<br />
I needed to re-create it for myself so I could understand what was going on. I started by setting my sign-in cookie to expire after 10 seconds on my login page.<br />
<br />
var authenticationManager = HttpContext.Current.GetOwinContext().Authentication;<br />
authenticationManager.SignIn(new AuthenticationProperties()<br />
{<br />
IsPersistent = chkRemember.Checked, // Tells OWIN whether or not to create a persistent cookie instead of just a session cookie<br />
IssuedUtc = DateTime.UtcNow,<br />
ExpiresUtc = DateTime.UtcNow.AddSeconds(10)<br />
}, userIdentity);<br />
<div>
<br /></div>
Next, in my Startup.cs I set my ExpireTimeSpan to 1 minute, and my validateInterval to 10 seconds. I'm not sure that ExpireTimeSpan actually does anything, it feels like it gets over-ridden by the ExpiresUtc from the SignIn code.<br />
<br />
var cookieExpiration = TimeSpan.FromMinutes(1);<br />
var checkPeriod = TimeSpan.FromSeconds(10);<br />
<br />
app.UseCookieAuthentication(new CookieAuthenticationOptions<br />
{<br />
...<br />
Provider = new CookieAuthenticationProvider<br />
{<br />
OnValidateIdentity = SecurityStampValidator.OnValidateIdentity<applicationusermanager applicationuser="">(</applicationusermanager><br />
validateInterval: checkPeriod,<br />
regenerateIdentity: (manager, user) => manager.CreateIdentityAsync(user, DefaultAuthenticationTypes.ApplicationCookie)<br />
),<br />
},<br />
ExpireTimeSpan = cookieExpiration,<br />
SlidingExpiration = true,<br />
});<br />
<div>
<br /></div>
Once I had these very short times in place I started up the site. By first logging in and then, after 5 seconds had passed and before the 10 second mark, navigating to the root of the site ( "/" ) I was able to consistently generate the error. Every 5 seconds or so if I went to the root I would throw that error. As expected, the user never actually sees the error, it just gets thrown in back end code, so any logging you have setup should catch it.<br />
<br />
<b>Cause</b><br />
<br />
If you did not read any of the articles linked to above, this error is caused by two things. First, IIS in integrated mode will execute the .net code twice when the site root is requested. First it starts what is called the "parent" call, but immediately sees that the request is for root, which does not exist, so it starts a "child" call to whatever the default document is. This child call executes correctly and is prepped for return to the client. When the child call is finished, the parent call resumes executing, only now the Response has already been created and prepared for the client, and this is where the Owin bugs come into play.<br />
<br />
Because the Owin code does not check before attempting to set header information in the default <a href="https://github.com/aspnet/AspNetKatana/blob/e2b18ec84ceab7ffa29d80d89429c9988ab40144/src/Microsoft.Owin/Infrastructure/ChunkingCookieManager.cs">ChunkingCookieManager</a>, it blows up during this parent call as Owin is trying to write out cookie information to the header. This bug actually exists at least twice in Owin, so if you fix the first one, then the second one still blows up with the same error just in a different place.<br />
<br />
The bad news is that there is no good or easy way to deal with this error. The good news is that there is a way of wrapping the first error so that your error handler can uniquely identify and handle it with minimal effort.<br />
<br />
<b>Solution 1:</b><br />
<br />
In Global.asax.cs in the Application_Error method, where you are likely trapping the errors, you can inspect the error description as well as its stack trace and make a decision on how to handle it that way.<br />
<br />
protected void Application_Error(object sender, EventArgs e)<br />
<div>
<div>
Exception parent_ex = Server.GetLastError();</div>
<div>
Exception ex = parent_ex.GetBaseException();</div>
</div>
<div>
if(ex.Message.StartsWith("Server cannot append header after HTTP headers have been sent.") && ex.StackTrace.Contains("Microsoft.Owin.Infrastructure.ChunkingCookieManager.AppendResponseCookie")) then do something</div>
<br />
<b>Solution 2:</b><br />
<br />
While neither of these solutions is great, and solution 1 is easier, it also feels messier. The second solution is to wrap the CookieManager and catch and wrap the exception there. This feels cleaner because it is much closer to the source, unfortunately you cannot handle the error there or the next error will just crop up.<br />
<br />
replace the default CookieManager with your custom wrapper:<br />
<br />
app.UseCookieAuthentication(new CookieAuthenticationOptions<br />
{<br />
...<br />
CookieManager = new ChunkingCookieManagerHeaderSentCheck(),<br />
...<br />
<div>
<br /></div>
<div>
<br /></div>
<div>
then, for the most part, just wrap the default ChunkingCookieManager. However, in the problem method insert a try/catch that provides a bit more detail. You still have to look for this detail in Global.asax.cs, but it is better than trying to comb through a stack trace. You could also use a custom exception here with extra fields that could help you in identifying this specific exception.</div>
<div>
<br /></div>
<div>
<div>
public class ChunkingCookieManagerHeaderSentCheck : ICookieManager</div>
<div>
{</div>
<div>
private readonly ChunkingCookieManager _chunkingCookieManager;</div>
<div>
<br /></div>
<div>
public ChunkingCookieManagerHeaderSentCheck()</div>
<div>
{</div>
<div>
_chunkingCookieManager = new ChunkingCookieManager();</div>
<div>
}</div>
<div>
<br /></div>
<div>
public string GetRequestCookie(IOwinContext context, string key)</div>
<div>
{</div>
<div>
return _chunkingCookieManager.GetRequestCookie(context, key);</div>
<div>
}</div>
<div>
<br /></div>
<div>
public void AppendResponseCookie(IOwinContext context, string key, string value, CookieOptions options)</div>
<div>
{</div>
<div>
/*</div>
<div>
* Microsof's Chunking Cookie Manager has a bug in it where it does not check to see</div>
<div>
* if headers have already been sent before it attempts to set more.</div>
<div>
* The exception cannot be dealt with here because the same bug exists in</div>
<div>
* CookieAuthenticationHandler.ApplyResponseGrantAsync.</div>
<div>
* Since it is harder to catch that one, it is best to just let this one be thrown,</div>
<div>
* but add some explanation to it so error handling can single it out in Global.asax.cs</div>
<div>
* and deal with it appropriately.</div>
<div>
*/</div>
<div>
try</div>
<div>
{</div>
<div>
_chunkingCookieManager.AppendResponseCookie(context, key, value, options);</div>
<div>
}</div>
<div>
catch (Exception ex)</div>
<div>
{</div>
<div>
if (context.Request.Uri.AbsolutePath == "/" && ex.Message.StartsWith("Server cannot append header after HTTP headers have been sent"))</div>
<div>
throw new Exception("Owin CookieManager attempted to write headers to root response.", ex);</div>
<div>
else</div>
<div>
throw;</div>
<div>
}</div>
<div>
}</div>
<div>
<br /></div>
<div>
public void DeleteCookie(IOwinContext context, string key, CookieOptions options)</div>
<div>
{</div>
<div>
_chunkingCookieManager.DeleteCookie(context, key, options);</div>
<div>
}</div>
<div>
}</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>Conclusion</b></div>
<div>
<br /></div>
<div>
Hopefully Microsoft will fix their code some day. However it has been a known issue since June 2017, and so far they have only made excuses for why they do not feel like fixing it. So until it actually makes it onto their road map, these appear to be the best options for dealing with the issue.</div>
<div>
<br /></div>
<div>
As a quick note. I read that Server.Transfer and Server.Execute also trigger this parent/child request feature of IIS. If that is true, then they should also trigger this bug. However, in my testing with Server.Transfer, I was not able to trip this bug.</div>
<br />Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com2tag:blogger.com,1999:blog-703101161281072462.post-52268184943152519242019-05-22T10:57:00.000-07:002019-05-22T10:57:37.688-07:00OWASP ZAP APIWhen testing a web application there are many different types of tests you can run and automate, and many different ways of doing so. I currently work in an environment where Microsoft's Team Foundation Server (now called Azure DevOps Server) runs a series of Unit tests and Selenium tests in an automated fashion. We initially used CodedUI until Microsoft discontinued it, then we converted our code base.<br />
<br />
This setup is one giant collection of regression tests like every project should have. And mingled in with them was some basic security testing from a users perspective as well. However, what we did not have was a method of automating the ability to test hack attempts against our sites. One of the reasons why is the shear number of possible hacks and combination hacks that are possible and should be tested for.<br />
<br />
Enter the <a href="https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project">OWASP ZAP tool</a>. A bunch of security experts have formed a non profit to educate people about security, and one of that companies outputs has been a free (and well maintained) tool for attacking your own sites and producing reports about any security issues it finds. The best part is that this tool has a pretty well developed REST API, so you can run it in an automated fashion.<br />
<br />
The ZAP tool has a decent API UI to help you learn it, but other than that is lacking documentation for it, which is the reason for this post.<br />
<br />
If you open the ZAP tool GUI, click on Tools / Options / API you can find your API key and a few API related settings you can mess with. By default, the tool has a web UI at http://localhost:8080/. Clicking on the Local API link inside of that will get you into the REST API help and demo area where you can run each call.<br />
<br />
In the GUI you can enter the base URL for your site, and click Attack. That is an easy method of running a one-off attack against your site. But if you want to automate this process, there are several REST calls involved.<br />
<br />
The first thing you need to do is start the application via command line. On windows it installs to:<br />C:\Program Files\OWASP\Zed Attack Proxy\zap.bat<br />
<br />
Then call these REST endpoints:<br />
- NewSession (clear all unsaved scan histories, just for a fresh start; you could load saved sessions instead if desired)<br />
http://localhost:8080/JSON/core/action/newSession/?zapapiformat=JSON&apikey=[yourapikey]<br />
- SetMode ( sets the scan mode to attack level )<br />
http://localhost:8080/JSON/core/action/setMode/?zapapiformat=JSON&apikey=[yourapikey]&mode=attack<br />
- Spider.Scan ( spider the url to find all the links to attack, this has to be performed before an Ascan attack is performed, the attack itself is only done on URLs already in the ZAP session. The Spider is the most efficient method of crawling one or more pages and loading all the found URLs into memory. The attack method only attacks distinct URLs, so it does not matter if the spider duplicates URLs while it is crawling pages. )<br />
http://localhost:8080/JSON/spider/action/scan/?zapapiformat=JSON&apikey=[yourapikey]&url=[an escaped url to crawl]&recurse=true<br />
- AScan.Scan ( fire off an attack for all links the spider found under this base URL )<br />
http://localhost:8080/JSON/ascan/action/scan/?zapapiformat=JSON&apikey=[yourapikey]&url=[an escaped url to lookup]&recurse=true<br />
- JsonReport ( get the total report for all scans and attacks in json format )<br />
http://localhost:8080/OTHER/core/other/jsonreport/?apikey=[yourapikey]<br />
<div>
<br /></div>
<div>
There are many more options, and other REST endpoints you can call to customize from there and to perform other and more detailed automated attacks. However, those are the base required to create an automated scan and attack. Hopefully that provides enough of a jump start so others can get the concept down and process further on their own.</div>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-5388743085909212312019-01-29T10:00:00.000-08:002019-01-29T11:04:59.303-08:00Installing Milestone Xprotect +I have been a long time fan of the Milestone XProtect NVR software. I have multiple installations, and have fought through many install and upgrade issues.<br />
<br />
Before the Plus line of their software came out the XProtect product was pretty stable. It just worked, albeit a bit limited in features. The new Plus line fixes all the feature issues (lots of fun new features), but the install and upgrade process has become extremely fragile in the process. Hopefully over time they will start to fix some of these issues.<br />
<br />
In the meantime, I thought I would create an install list for anyone else struggling with it. Almost all of my installs are on smaller windows 10 servers. Considering that is not even a server OS, it was an afterthought by Milestone when they first started releasing the new product line, and might be the cause of some of these problems.<br />
<br />
Install Steps:<br />
- Uninstall IIS (Milestone will automatically install and configure IIS, if you already have it installed then sometimes the install will fail, especially if you have changed any of the default settings).<br />
- <strike>Install SQL 2016 Express. Milestone does not yet support later versions of SQL, and SQL 2015 that it ships with has a random bug in the GEO portion of its code that can cause the Milestone installation to fail. While technically a SQL bug, Milestone could easily fix this by simply shipping sql 2016 with their product instead of 2015.</strike><br />
( I have been informed by Milestone that as of version 2018, SQL 2016 is the default shipped with it. )<br />
- Do not at any point re-name the computer this is being installed on. There is a Milestone/SQL bug where Milestone will get the old computer name from SQL and fail during install if you do this. Again, a SQL bug, but one that Milestone could solve. If you really want to rename it, then I have heard that there is a SQL script you can run so SQL sees the new computer name. Research renaming a sql server computer name.<br />
- Next start the Milestone installation, however, do a custom installation and only install the Recording server (assuming you are planning on an all-in-one box). If you try and install all the software at once then it generally fails. I believe this is due to a security bug in the Mobile Server install, but am not sure.<br />
- Now do another Custom Milestone install this time installing everything. Again, the custom install will prevent the Mobile server from being automatically installed.<br />
- Open up the Management Client and create a new local admin account. XProtect has some security bugs I have not yet figured out related to using the build in windows accounts which is the default.<br />
- Use this local admin account when connecting to the Smart Client if the build-in account does not work. Not a bad habit anyway in case you log into the computer with a different account in the future.<br />
- Find and install the Milestone Mobile Server package. When it asks for credentials to connect to the recording server, use the new local admin account you created instead of the default of the windows credentials.<br />
<br />
At this point you should be up and running with a new install of Milestone XProtect+. I have run into issues during upgrades where the database got corrupted in some fashion so everything looked good, but then I started having quirky issues when messing around with trying to bring cameras back online.Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-65669576816571754692018-05-11T12:20:00.001-07:002018-05-11T12:22:02.860-07:00Pacific Power Hourly Energy SavingsOn the North West coast of the United States is a company called Pacific Power. They are the government approved power provider.<br />
<br />
There is a special program they have that purports to incentivize people to shift their power usage to off peak periods. This is a very good idea to help balance out the power grid, and I am a huge fan of the idea. However, it does cause a great deal of work for those who shift their usage and they do deserve to be incentivized to do so.<br />
<br />
Unfortunately, it turns out that the program Pacific Power runs is a scam. Their documentation, and their representatives who try and explain the documentation, claims that the normal household is charged 3 cents per kilowatt hour. Once switched to the new program the household will be charged 6 cents per kilowatt hour for on peak usage, and only 1 cent per kilowatt hour for off peak usage. This seems reasonable, and a hard working person should be able to save some money by changing their habits fairly drastically.<br />
( One of the documents actually says it will be a negative 1 cent, the reps just seem to explain it poorly )<br />
<br />
The reality of this program is far different from what is advertised.<br />
<br />
In reality, Pacific Power does not stop charging the household for anything, they continue charging the standard electrical fees, and then they tack on an ADDITIONAL 6 cents per kilowatt hour for peak usage, and a -1 cent per kilowatt hour for off peak usage.<br />
<br />
So, by completely revamping your lifestyle, you might save $1-2 a month if you are lucky, if you slip then you will suddenly be paying a massively inflated bill on top of the normal electrical bill you were already paying.Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-47735491590081713092018-03-16T16:58:00.000-07:002018-05-11T12:22:17.921-07:00Microsoft TFS Server Custom SQL Query<div class="tr_bq">
Microsoft's TFS 2017 has come a long ways, but is still a pretty terrible product. That said, it is still the best option for many things as a Microsoft developer.</div>
<br />
One thing I have seen a lot of people on the Internet asking about is various ways to get at the TFS information stored in SQL. Microsoft obviously refuses to answer these questions just pointing people to their APIs. However, sometimes it is just easier to run a quick SQL query.<br />
<br />
For those people who are just wondering where Microsoft is hiding all the various fields, I have spent a good bit of time tracking some of them down. Remember this code is for SQL 2017 and is unlikely to work for other versions, but it might point you in the right direction. All this is is one of my queries with some comments, I will leave it up to you to go look at the tables and see how I use them to get the data.<br />
<br />
The very last join I do is for a custom HTML field that I created in TFS, a very similar join on that table could be used to get other HTML fields as well, such as comments or description fields.<br />
<blockquote>
// Dashboards.tbl_Widget stores the Dashboard settings, you can hack changes here<br />
// dbo.QueryItems has the settings for each query<br />
// tbl_ChangeSet stores the change sets<br />
// vw_WorkItemComments stores all the work item comments, including comments generated when changesets are submitted<br />
// tbl_Version holds actual file references, can be linked to changesets using VersionFrom column<br />
// WorkItemFiles are links between workitems and the changesets<br />
// dbo.WorkItemLongTexts stores the long texts from bugs, like the main body<br />
// dbo.tbl_TagDefinition contains the tag names themselves joins dbo.tbl_PropertyDefinition.Name on TagId<br />
// dbo.tbl_Field contains the descriptions for the FieldId's from tbl_WorkItemCustomLatest<br />
// dbo.tbl_PropertyDefinition string guid table join on dbo.tbl_PropertyValue on PropertyId integer value, ArtifactId is lookup key which is WorkItemId number converted to hex</blockquote>
<br />
<blockquote>
select [System.Id] ID, l_title.[TextValue] [Title], l_approvedby.[StringValue] [ApprovedBy], l_valuearea.[StringValue] [ValueArea], l_priority.[IntValue] [Priority], l_waitingon.StringValue [WaitingOn]<br />
<span style="white-space: pre;"> </span>-- Sub Select for Tags<br />
<span style="white-space: pre;"> </span>,(select substring((select distinct ', ' + td.[Name]<br />
<span style="white-space: pre;"> </span>from dbo.tbl_PropertyValue pv<br />
<span style="white-space: pre;"> </span>Inner Join dbo.tbl_PropertyDefinition pd on pv.PropertyId = pd.PropertyId<br />
<span style="white-space: pre;"> </span>Inner Join dbo.tbl_TagDefinition td on pd.[Name] = 'Microsoft.TeamFoundation.Tagging.TagDefinition.' + cast(td.TagId as varchar(100))<br />
Where pv.InternalKindId = 16 And pv.ArtifactId = CONVERT(VARBINARY(8), l.[System.Id])<br />
<span style="white-space: pre;"> </span>for xml path('')), 3, 9000)) as Tags<br />
<span style="white-space: pre;"> </span>,l_deploy.Words DeployNotes<br />
from vw_denorm_WorkItemCoreLatest l<br />
inner join vw_denorm_WorkItemCustomLatest l_title on l.[System.Id] = l_title.Id And l_title.FieldId = 1<br />
left join vw_denorm_WorkItemCustomLatest l_approvedby on l.[System.Id] = l_approvedby.Id And l_approvedby.FieldId = 10133<br />
<span style="white-space: pre;"> </span>left join vw_denorm_WorkItemCustomLatest l_valuearea on l.[System.Id] = l_valuearea.Id And l_valuearea.FieldId = 10055<br />
left join vw_denorm_WorkItemCustomLatest l_priority on l.[System.Id] = l_priority.Id And l_priority.FieldId = 10029<br />
left join vw_denorm_WorkItemCustomLatest l_waitingon on l.[System.Id] = l_waitingon.Id And l_waitingon.FieldId = 10134<br />
left join (select Max(AddedDate) AddedDate, Id from WorkItemLongTexts Where FldID = 11135 Group By Id ) l_deploy_grp on l.[System.Id] = l_deploy_grp.Id<br />
<span style="white-space: pre;"> </span>left join WorkItemLongTexts l_deploy on l.[System.Id] = l_deploy.Id And l_deploy.FldID = 11135 And l_deploy_grp.AddedDate = l_deploy.AddedDate<br />
Where<br />
<span style="white-space: pre;"> </span>l.[System.IsDeleted] = 0<br />
And l.[System.State] = '20 Testing'<br />
And l.[System.AreaPath] = '\VerizonRAB'<br />
<span style="white-space: pre;"> </span>And ( l.[System.WorkItemType] = 'Feature' Or l.[System.WorkItemType] = 'Bug' Or l.[System.WorkItemType] = 'Task' )</blockquote>
<br />
<br />
<br />Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-22343423764665132082018-01-03T09:46:00.000-08:002018-05-11T12:22:29.886-07:00WatchGuard FireBox Phone VPN ConfigurationWe recently migrated from a very old Cisco ASA firewall to a couple of WatchGuard FireBox M200 devices. So far my assessment is that they are very capable devices with lots of features, but inadequately documented and the company has poor support.<br />
<br />
Part of my negative experience in regards to support may be from buying through a third party re-seller. WatchGuard does not want to talk to you directly, they want to talk to your re-seller.<br />
<br />
Most of the configuration and setup went fairly smoothly. Due to poor documentation there were some gotcha's when setting up active directory authentication in regards to case sensitivity. The little documentation WatchGuard does have is well formatted and reads nicely. Unfortunately, due to there being so little of it you end up trying to reference the wrong documentation to help fill in the blanks for missing documentation. Many companies try and fill this gap by providing help forums that they monitor and respond to questions on which then fills in all the missing information. WatchGuard seems to be missing a good implementation of this, and they do not seem to respond when people reach out to them.<br />
<br />
You can see a perfect example of all of WatchGuard's issues by looking at their android vpn app in the Google Play store. The app is poorly written, functions horribly, and they ignore all the people providing negative feedback who are essentially begging for help.<br />
<br />
Towards the end of my own WatchGuard configuration I intended to setup an IPSec configuration for phones to connect to as indicated in their documentation. Unfortunately their own phone vpn client is utterly worthless leaving you to try and find something else.<br />
<br />
I did use WatchGuard's iPhone configuration instructions to get the native Android VPN client to connect. Unfortunately it ended up having two major issues that caused me to dump it. The first is that WatchGuard has a limitation or perhaps a bug in it's IPSec implementation for Active Directory authentication. As near as I can tell, either SSL or IPSec can authenticate via Active Directory but not both. It is possible that by deleting the SSL config, setting up IPSec, then re-creating the SSL config I could get around this bug, but it was not worth attempting at the moment. When using AD authentication in the configuration I had setup, it would connect for a few seconds, then disconnect. Which tells me there must me some sort of timing mismatch. You would think that would be simple to adjust and overcome, but again the lack of documentation means I either have to pay for support or just deal with it for now.<br />
<br />
The second problem, is that even when using the WatchGuard database native authentication, the native client on android does not seem to hold in a connection when the phone goes to sleep. So if I set my phone down for a few minutes and come back to it I have to re-connect every time.<br />
<br />
Fortunately WatchGuard does support OpenVPN. So I was able to download a third party client that works with the existing SSL configuration successfully. Not ideal, but it works for now.<br />
<br />
-<span style="white-space: pre;"> </span>1: download the client.ovpn attachment to your phone. You can find this by going to the ip address of your firewall in a browser, authenticate, and download the "Mobile VPN with SSL client profile".<br />
-<span style="white-space: pre;"> </span>2: Open up the App store on your respective device and find the app OpenVPN Connect (not just OpenVPN, it has to say Connect).<br />
o<span style="white-space: pre;"> </span>Install the app and open it.<br />
-<span style="white-space: pre;"> </span>3: Click on the menu in the upper right, the three dots.<br />
-<span style="white-space: pre;"> </span>4: Click the menu item Import Profile from SD card.<br />
-<span style="white-space: pre;"> </span>5: navigate to the folder containing the client.ovpn attachment that you downloaded in step 1. For me it was in my Download folder.<br />
-<span style="white-space: pre;"> </span>6: Click on the client.ovpn file and click the Select button. It may be confusing because the client.ovpn line you selected will not highlight.<br />
-<span style="white-space: pre;"> </span>7: If you chose the correct file you should now see an IP address in the OpenVPN Profile area.<br />
o<span style="white-space: pre;"> </span>Enter your Active Directory username and password into the provided fields.<br />
o<span style="white-space: pre;"> </span>Click Connect<br />
<div>
<br /></div>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-82897114757879873192017-12-11T19:57:00.004-08:002017-12-12T07:36:50.079-08:00Controling SOAP security header “mustUnderstand” attribute in WCF client<div style="background-color: white; border: 0px; clear: both; color: #242729; font-family: Arial, "Helvetica Neue", Helvetica, sans-serif; font-size: 15px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; padding: 0px; vertical-align: baseline;">
I was recent tasked with connecting .NET WCF with an older java soap service. Unfortunately the service was not standard and needed more WCF customization than normal, starting with the security.</div>
<div style="background-color: white; border: 0px; clear: both; color: #242729; font-family: Arial, "Helvetica Neue", Helvetica, sans-serif; font-size: 15px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; padding: 0px; vertical-align: baseline;">
I needed to include both a client x509 certificate for the Transport tunnel, and a username/password combination in the soap header itself. By playing around with the web.config binding configurations I could have eventually achieved this. However that was not the end of the customizations it needed.</div>
<div style="background-color: white; border: 0px; clear: both; color: #242729; font-family: Arial, "Helvetica Neue", Helvetica, sans-serif; font-size: 15px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; padding: 0px; vertical-align: baseline;">
This service needed the EncodedMustUnderstand flag set to false for the security header. Back with WSE that used to be a very simple flag to set, but for some reason Microsoft decided to remove that capability from WCF. One guy claimed to find a <a href="https://stackoverflow.com/questions/4169798/modify-soap-header-mustunderstand-attribute-in-wcf-client">simple fix for this</a>, however his fix did not work for me.</div>
<div style="background-color: white; border: 0px; clear: both; color: #242729; font-family: Arial, "Helvetica Neue", Helvetica, sans-serif; font-size: 15px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; padding: 0px; vertical-align: baseline;">
Throughout the years since WCF has been introduced many people have had this or very similar issues when dealing with Java based SOAP services. I have yet to find someone who has discovered a good solution. In my own two week journey to solving this issue I had to do some pretty serious customization to WCF. By sharing my discoveries here I hope to help other people shorted the time they spend enduring the same pain.</div>
<div style="background-color: white; border: 0px; clear: both; color: #242729; font-family: Arial, "Helvetica Neue", Helvetica, sans-serif; font-size: 15px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; padding: 0px; vertical-align: baseline;">
The first thing I did was to create an extension method for the service allowing me to create a CustomBinding for it. This method served as the hub to which I attached various other classes and methods as I built them. It seems daunting, and it really is. Separating out the various ideas into their own files was the approach I took to keeping the major areas of functionality separate in my head.<br />
<br />
<!-- HTML generated using hilite.me --><b>Base Helper Class</b><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Configuration</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Channels</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Security</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Security.Tokens</span>;
<span style="color: #008800; font-weight: bold;">namespace</span> <span style="color: #0e84b5; font-weight: bold;">Services</span>
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">static</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">ClientHelper</span>
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">static</span> PClient <span style="color: #0066bb; font-weight: bold;">Initialize</span>(<span style="color: #008800; font-weight: bold;">this</span> PClient client)
{
<span style="color: #333399; font-weight: bold;">var</span> urlConfig = ConfigurationManager.AppSettings[<span style="background-color: #fff0f0;">"Url"</span>].ToString();
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #333399; font-weight: bold;">string</span>.IsNullOrEmpty(urlConfig))
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">Exception</span>(<span style="background-color: #fff0f0;">"Missing Url Config"</span>);
<span style="color: #333399; font-weight: bold;">var</span> url = <span style="color: #008800; font-weight: bold;">new</span> UrlSecurityConfig(urlConfig);
<span style="color: #888888;">//System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;</span>
<span style="color: #333399; font-weight: bold;">var</span> security = SecurityBindingElement.CreateCertificateOverTransportBindingElement();
security.IncludeTimestamp = <span style="color: #008800; font-weight: bold;">true</span>;
security.DefaultAlgorithmSuite = SecurityAlgorithmSuite.Basic256;
security.MessageSecurityVersion = MessageSecurityVersion.WSSecurity11WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10;
security.EndpointSupportingTokenParameters.Signed.Add(<span style="color: #008800; font-weight: bold;">new</span> UserNameSecurityTokenParameters()); <span style="color: #888888;">// add specific username security feature</span>
security.SecurityHeaderLayout = SecurityHeaderLayout.Lax;
security.EnableUnsecuredResponse = <span style="color: #008800; font-weight: bold;">true</span>;
<span style="color: #888888;">//security.MessageSecurityVersion = MessageSecurityVersion.WSSecurity11WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11;</span>
<span style="color: #888888;">//security.EndpointSupportingTokenParameters.SignedEncrypted.Add(new X509SecurityTokenParameters(X509KeyIdentifierClauseType.Any, SecurityTokenInclusionMode.AlwaysToRecipient)); // add specific x509 cert security security.EndpointSupportingTokenParameters.Signed.Add(new UserNameSecurityTokenParameters()); // add specific username security feature</span>
<span style="color: #888888;">//security.DefaultAlgorithmSuite = new Basic128Sha256Rsa15Sha1AlgorithmSuite(); // when we need to tweak the security suite</span>
<span style="color: #888888;">//var encoding = new TextMessageEncodingBindingElement();</span>
<span style="color: #888888;">//encoding.MessageVersion = MessageVersion.CreateVersion(EnvelopeVersion.Soap11, AddressingVersion.None);// MessageVersion.Soap11;</span>
<span style="color: #888888;">////encoding.MessageVersion = MessageVersion.Soap11WSAddressingAugust2004;</span>
<span style="color: #888888;">//encoding.WriteEncoding = Encoding.UTF8;</span>
<span style="color: #333399; font-weight: bold;">var</span> transport = <span style="color: #008800; font-weight: bold;">new</span> HttpsTransportBindingElement();
transport.MaxReceivedMessageSize = <span style="color: #6600ee; font-weight: bold;">20000000</span>; <span style="color: #888888;">// 20 megs</span>
transport.RequireClientCertificate = <span style="color: #008800; font-weight: bold;">false</span>;
transport.TransferMode = TransferMode.Buffered;
transport.DecompressionEnabled = <span style="color: #008800; font-weight: bold;">false</span>;
CustomBinding binding = <span style="color: #008800; font-weight: bold;">new</span> CustomBinding();
binding.Elements.Add(security);
<span style="color: #888888;">//binding.Elements.Add(encoding); // add normal encoding</span>
binding.Elements.Add(<span style="color: #008800; font-weight: bold;">new</span> MustUnderstandOffBindingElement(<span style="background-color: #fff0f0;">"UTF-8"</span>, <span style="background-color: #fff0f0;">"text/xml"</span>, MessageVersion.Soap11)); <span style="color: #888888;">// add custom encoder</span>
binding.Elements.Add(transport);
<span style="color: #333399; font-weight: bold;">var</span> x509Config = ConfigurationManager.AppSettings[<span style="background-color: #fff0f0;">"x509"</span>].ToString();
<span style="color: #333399; font-weight: bold;">var</span> X509Cert = Common.Get509Cert(x509Config);
<span style="color: #888888;">// THIS IS WHEN THE CERT NEEDS A SPECIFIC DOMAIN IDENTITY SPECIFIED</span>
<span style="color: #888888;">//var identity = EndpointIdentity.CreateX509CertificateIdentity(X509Cert);</span>
<span style="color: #888888;">//var address = new EndpointAddress(new Uri(url), identity);</span>
<span style="color: #888888;">////var binding = new WSHttpBinding(SecurityMode.Message);</span>
<span style="color: #888888;">////binding.Security.Message.ClientCredentialType = MessageCredentialType.Certificate;</span>
<span style="color: #888888;">//var factory = new ChannelFactory<CoAServices_v1r21_P>(binding, address);</span>
<span style="color: #888888;">////factory.Endpoint.EndpointBehaviors.Remove(typeof(System.ServiceModel.Description.ClientCredentials));</span>
<span style="color: #888888;">////factory.Endpoint.EndpointBehaviors.Add(new Services.WsNonceCustomCredentials());</span>
<span style="color: #888888;">//factory.Credentials.UserName.UserName = username;</span>
<span style="color: #888888;">//factory.Credentials.UserName.Password = password;</span>
<span style="color: #888888;">//factory.Credentials.ClientCertificate.Certificate = X509Cert;</span>
<span style="color: #888888;">//factory.Credentials.ServiceCertificate.DefaultCertificate = X509Cert;</span>
client.Endpoint.Binding = binding;
client.Endpoint.Address = <span style="color: #008800; font-weight: bold;">new</span> EndpointAddress(url.Url);
<span style="color: #888888;">//client.Endpoint.EndpointBehaviors.Add(new ExClientBehavior()); // adding a custom behavior built in a custom class</span>
<span style="color: #888888;">//client.Endpoint.EndpointBehaviors.Add(new MustUnderstandBehavior(false)); // create a behavior with the MustUnderstand attribute set to false, does not work for headers</span>
<span style="color: #888888;">//// replace normal credencials with custom ones that generate Nonce:</span>
<span style="color: #888888;">//client.ChannelFactory.Endpoint.Behaviors.Remove<System.ServiceModel.Description.ClientCredentials>();</span>
<span style="color: #888888;">//client.ChannelFactory.Endpoint.Behaviors.Add(new Services.WsNonceCustomCredentials());</span>
client.ClientCredentials.UserName.UserName = url.UserName;
client.ClientCredentials.UserName.Password = url.Password;
client.ClientCredentials.ClientCertificate.Certificate = X509Cert;
<span style="color: #888888;">//client.ClientCredentials.ServiceCertificate.DefaultCertificate = X509Cert;</span>
<span style="color: #888888;">//client.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.None;</span>
<span style="color: #888888;">//client.ClientCredentials.ServiceCertificate.Authentication.RevocationMode = X509RevocationMode.NoCheck;</span>
<span style="color: #008800; font-weight: bold;">return</span> client;
<span style="color: #888888;">//return factory.CreateChannel();</span>
}
}
}
</pre>
</div>
<br />
<br />
As you can see, there is lots of commented out code to plug in various pieces of functionality. I ended up not needing half of it, but there was no point in throwing away code that functioned. For the purposes of this post, only the commented out ExClientBehavior lines and the custom Nonce credencials will be discussed in addition to the working lines of code.<br />
<br />
One of the first steps I took was to create a helper class with a method to get my x509 certificate. I later added a method to do some basic url config parsing. This ended up being the easiest part of the whole process.<br />
<br />
EDIT: A quick side note, when generating the code below for this blog post, it kept wanting to rename X509Certificate2Collection and other class names as <span style="color: #333333;">17bH1SYLoBdGsBaDedPR2EE3JUt8oRS7qd. A very odd quirk, but I have seen that weird name in several Microsoft posts as well, so apparently I am not the only one who has had that issue when writing posts. </span><br />
<span style="color: #333333;"><br /></span>
<span style="color: #333333;">EDIT2: It turns out the weird renaming was caused by a Chrome extension I am running called uBlock. So if you see something similar, then check your extensions. In my case, white listing the blogger domain fixed the issue.</span><br />
<br />
<!-- HTML generated using hilite.me --><b>Common Helper Methods</b><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Linq</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Security.Cryptography.X509Certificates</span>;
<span style="color: #008800; font-weight: bold;">namespace</span> <span style="color: #0e84b5; font-weight: bold;">Services</span>
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">static</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">Common</span>
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">static</span> X509Certificate2 <span style="color: #0066bb; font-weight: bold;">Get509Cert</span>(<span style="color: #333399; font-weight: bold;">string</span> LocationConfig)
{
<span style="color: #333399; font-weight: bold;">var</span> locationParts = LocationConfig.Split(<span style="color: #0044dd;">':'</span>);
<span style="color: #008800; font-weight: bold;">if</span> (locationParts.Length < <span style="color: #6600ee; font-weight: bold;">2</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">Exception</span>(<span style="background-color: #fff0f0;">"Invalid X509 Config"</span>);
X509Certificate2 cer = <span style="color: #008800; font-weight: bold;">new</span> X509Certificate2();
X509Store store = locationParts[<span style="color: #6600ee; font-weight: bold;">0</span>] == <span style="background-color: #fff0f0;">"CurrentUser"</span> ? <span style="color: #008800; font-weight: bold;">new</span> X509Store(StoreLocation.CurrentUser) : <span style="color: #008800; font-weight: bold;">new</span> X509Store(StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadOnly);
X509Certificate2Collection cers = store.Certificates.Find(X509FindType.FindBySubjectName, locationParts[<span style="color: #6600ee; font-weight: bold;">1</span>], <span style="color: #008800; font-weight: bold;">false</span>);
<span style="color: #008800; font-weight: bold;">if</span> (cers.Count == <span style="color: #6600ee; font-weight: bold;">0</span>)
{
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">Exception</span>(<span style="background-color: #fff0f0;">"Can't find X509 Cert."</span>);
}
<span style="color: #333399; font-weight: bold;">var</span> cert = cers[<span style="color: #6600ee; font-weight: bold;">0</span>];
store.Close();
<span style="color: #008800; font-weight: bold;">return</span> cert;
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">UrlSecurityConfig</span>
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #333399; font-weight: bold;">string</span> Url { <span style="color: #008800; font-weight: bold;">get</span>; <span style="color: #008800; font-weight: bold;">protected</span> <span style="color: #008800; font-weight: bold;">set</span>; }
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #333399; font-weight: bold;">string</span> UserName { <span style="color: #008800; font-weight: bold;">get</span>; <span style="color: #008800; font-weight: bold;">protected</span> <span style="color: #008800; font-weight: bold;">set</span>; }
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #333399; font-weight: bold;">string</span> Password { <span style="color: #008800; font-weight: bold;">get</span>; <span style="color: #008800; font-weight: bold;">protected</span> <span style="color: #008800; font-weight: bold;">set</span>; }
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">UrlSecurityConfig</span>(<span style="color: #333399; font-weight: bold;">string</span> config)
{
<span style="color: #008800; font-weight: bold;">if</span>(!config.Contains(<span style="color: #0044dd;">'@'</span>))
{
Url = config;
<span style="color: #008800; font-weight: bold;">return</span>;
}
<span style="color: #333399; font-weight: bold;">var</span> uendStart = config.LastIndexOf(<span style="color: #0044dd;">'@'</span>);
<span style="color: #333399; font-weight: bold;">var</span> uStart = config.Substring(<span style="color: #6600ee; font-weight: bold;">0</span>, config.IndexOf(<span style="color: #0044dd;">'/'</span>) + <span style="color: #6600ee; font-weight: bold;">2</span>);
<span style="color: #333399; font-weight: bold;">var</span> uEnd = config.Substring(uendStart + <span style="color: #6600ee; font-weight: bold;">1</span>);
Url = uStart + uEnd;
<span style="color: #333399; font-weight: bold;">var</span> nameEnd = config.IndexOf(<span style="color: #0044dd;">':'</span>, uStart.Length + <span style="color: #6600ee; font-weight: bold;">1</span>);
UserName = config.Substring(uStart.Length, nameEnd - uStart.Length);
Password = config.Substring(nameEnd + <span style="color: #6600ee; font-weight: bold;">1</span>, uendStart - nameEnd - <span style="color: #6600ee; font-weight: bold;">1</span>);
}
}
}
</pre>
</div>
<br />
After trying all sorts of ways to get into the security header using existing settings, I finally decided to write my own custom Behavior. This code I copied directly from someone else's post who was struggling with something very similar. Unfortunately, I got a rude shock when I found out that the security headers are not part of the request sent to the behavior.<br />
<br />
Without access to those I was unable to modify them, although in testing I discovered that I was unable to affect the output of the headers here anyway despite the code looking like it was working correctly when I stepped through it. Perhaps I still have something slightly off.<br />
<br />
<!-- HTML generated using hilite.me --><b>Custom Behavior Classes</b><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Channels</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Description</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Dispatcher</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Configuration</span>;
<span style="color: #008800; font-weight: bold;">namespace</span> <span style="color: #0e84b5; font-weight: bold;">Services</span>
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">ExClientBehavior</span> : IEndpointBehavior
{
<span style="color: #557799;">#region IEndpointBehavior Members</span>
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">void</span> <span style="color: #0066bb; font-weight: bold;">AddBindingParameters</span>(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
{
<span style="color: #888888;">// no op</span>
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">void</span> <span style="color: #0066bb; font-weight: bold;">ApplyClientBehavior</span>(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
{
ExInspector inspector = <span style="color: #008800; font-weight: bold;">new</span> ExInspector();
clientRuntime.MessageInspectors.Add(inspector);
<span style="color: #888888;">//clientRuntime.ValidateMustUnderstand = false;</span>
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">void</span> <span style="color: #0066bb; font-weight: bold;">ApplyDispatchBehavior</span>(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
{
<span style="color: #888888;">// no op</span>
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">void</span> <span style="color: #0066bb; font-weight: bold;">Validate</span>(ServiceEndpoint endpoint)
{
<span style="color: #888888;">// no op</span>
}
<span style="color: #557799;">#endregion</span>
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">ExClientBehaviorExtensionElement</span> : BehaviorExtensionElement
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> Type BehaviorType
{
<span style="color: #008800; font-weight: bold;">get</span> { <span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">typeof</span>(ExClientBehavior); }
}
<span style="color: #008800; font-weight: bold;">protected</span> <span style="color: #008800; font-weight: bold;">override</span> <span style="color: #333399; font-weight: bold;">object</span> <span style="color: #0066bb; font-weight: bold;">CreateBehavior</span>()
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ExClientBehavior</span>();
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">ExInspector</span> : IClientMessageInspector
{
<span style="color: #557799;">#region IClientMessageInspector Members</span>
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">void</span> <span style="color: #0066bb; font-weight: bold;">AfterReceiveReply</span>(<span style="color: #008800; font-weight: bold;">ref</span> Message reply, <span style="color: #333399; font-weight: bold;">object</span> correlationState)
{
<span style="color: #888888;">// no op</span>
<span style="color: #008800; font-weight: bold;">return</span>;
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #333399; font-weight: bold;">object</span> <span style="color: #0066bb; font-weight: bold;">BeforeSendRequest</span>(<span style="color: #008800; font-weight: bold;">ref</span> Message request, IClientChannel channel)
{
MessageBuffer buffer = request.CreateBufferedCopy(<span style="color: #333399; font-weight: bold;">int</span>.MaxValue);
Message newMessage = buffer.CreateMessage();
<span style="color: #008800; font-weight: bold;">while</span> (newMessage.Headers.Count > <span style="color: #6600ee; font-weight: bold;">0</span>)
newMessage.Headers.RemoveAt(<span style="color: #6600ee; font-weight: bold;">0</span>);
<span style="color: #008800; font-weight: bold;">foreach</span> (<span style="color: #333399; font-weight: bold;">var</span> info <span style="color: #008800; font-weight: bold;">in</span> request.Headers)
{
newMessage.Headers.Add(MessageHeader.CreateHeader
(
info.Name,
info.Namespace,
<span style="color: #333399; font-weight: bold;">string</span>.Empty,
<span style="color: #008800; font-weight: bold;">false</span>,
<span style="color: #333399; font-weight: bold;">string</span>.Empty,
info.Relay
)
);
}
request = newMessage;
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">null</span>;
}
<span style="color: #557799;">#endregion</span>
}
}
</pre>
</div>
<br />
My next approach was to try writing my own Custom Encoder. Many people had stated that was the way to get at the stream both as it was heading out the door, and as it was coming back in before it hit the service code. I copied Microsoft's code and directions from <a href="https://docs.microsoft.com/en-us/dotnet/framework/wcf/samples/custom-message-encoder-custom-text-encoder">here</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/system.servicemodel.channels.messageencoderfactory?view=netframework-4.7.1">here</a>, and <a href="https://docs.microsoft.com/en-us/dotnet/api/system.servicemodel.channels.messageencoder?view=netframework-4.7.1">here</a>.<br />
<br />
I immediately started running into problems due to that class name rewrite in Microsoft's web posts that I mentioned earlier. It took me a little while to create names that all worked and referenced each other correctly.<br />
<br />
I could have handled the pain it had taken to get this far, but it wasn't over yet. Microsoft's code has a flaw in it that resulted in a loop which consistently generated stack overflow errors. I later discovered that the error centered around the "<span style="background-color: #f9f9f9; color: #222222; font-family: "consolas" , "menlo" , "monaco" , "lucida console" , "liberation mono" , "dejavu sans mono" , "bitstream vera sans mono" , "courier new" , monospace , sans-serif; font-size: 14px; white-space: pre;">this.factory = factory;</span>" line. It generated an internal reference to the parent object that the parent then referenced causing a recursive infinite loop.<br />
<br />
If there had been any sort of decent error that would have been easy to find and work through. Unfortunately, these were the errors and Inner Exceptions that it kept generating:<br />
<br />
<blockquote class="tr_bq" style="border: 0px; clear: both; font-family: Arial, "Helvetica Neue", Helvetica, sans-serif; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; padding: 0px; vertical-align: baseline;">
Message: An error occurred while making the HTTP request to <a href="https://service/" rel="nofollow noreferrer" style="border: 0px; color: #005999; cursor: pointer; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-decoration-line: none; vertical-align: baseline;">https://service</a>. This could be due to the fact that the server certificate is not configured properly with HTTP.SYS in the HTTPS case. This could also be caused by a mismatch of the security binding between the client and the server.</blockquote>
<br />
<blockquote class="tr_bq" style="border: 0px; clear: both; font-family: Arial, "Helvetica Neue", Helvetica, sans-serif; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; padding: 0px; vertical-align: baseline;">
Message: The underlying connection was closed: An unexpected error occurred on a send.</blockquote>
<br />
<blockquote class="tr_bq" style="border: 0px; clear: both; font-family: Arial, "Helvetica Neue", Helvetica, sans-serif; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; padding: 0px; vertical-align: baseline;">
Message: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.</blockquote>
Stepping through the code did show me that a stack overflow was occurring, but it was not obvious that it was a recursive reference that was causing it. And, since my code had been copied from an official Microsoft website, my first instinct was to trust it and look elsewhere. So I spent a long time looking at the exceptions and trying to figure out what they could mean.<br />
<br />
During this time I spent a lot of time in Fiddler working with the returning soap message and tweaking it to see how it would affect .net. I discovered that it is actually possible to strip off the characters surrounding the soap envelope that trigger the HTTP.SYS error in .net. This allowed me to test the service even when sending from an HTTP source rather than the required HTTPS source.<br />
<br />
I finally ended up with this working code. You will notice that the primary difference from Microsoft's classes is the replacement for the "mustUnderstand" line, which was the hack I put in to toggle that flag as the message went out the door.<br />
<br />
<!-- HTML generated using hilite.me --><b>Custom Encoder Classes</b><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Text</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Channels</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Description</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.IO</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Xml</span>;
<span style="color: #008800; font-weight: bold;">namespace</span> <span style="color: #0e84b5; font-weight: bold;">Services</span>
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">MustUnderstandOffBindingElement</span> : MustUnderstandOffBindingElement, IWsdlExportExtension
{
<span style="color: #008800; font-weight: bold;">private</span> MessageVersion msgVersion;
<span style="color: #008800; font-weight: bold;">private</span> <span style="color: #333399; font-weight: bold;">string</span> mediaType;
<span style="color: #008800; font-weight: bold;">private</span> <span style="color: #333399; font-weight: bold;">string</span> encoding;
<span style="color: #008800; font-weight: bold;">private</span> XmlDictionaryReaderQuotas readerQuotas;
MustUnderstandOffBindingElement(MustUnderstandOffBindingElement binding)
: <span style="color: #008800; font-weight: bold;">this</span>(binding.Encoding, binding.MediaType, binding.MessageVersion)
{
<span style="color: #008800; font-weight: bold;">this</span>.readerQuotas = <span style="color: #008800; font-weight: bold;">new</span> XmlDictionaryReaderQuotas();
binding.ReaderQuotas.CopyTo(<span style="color: #008800; font-weight: bold;">this</span>.readerQuotas);
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">MustUnderstandOffBindingElement</span>(<span style="color: #333399; font-weight: bold;">string</span> encoding, <span style="color: #333399; font-weight: bold;">string</span> mediaType,
MessageVersion msgVersion)
{
<span style="color: #008800; font-weight: bold;">if</span> (encoding == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"encoding"</span>);
<span style="color: #008800; font-weight: bold;">if</span> (mediaType == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"mediaType"</span>);
<span style="color: #008800; font-weight: bold;">if</span> (msgVersion == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"msgVersion"</span>);
<span style="color: #008800; font-weight: bold;">this</span>.msgVersion = msgVersion;
<span style="color: #008800; font-weight: bold;">this</span>.mediaType = mediaType;
<span style="color: #008800; font-weight: bold;">this</span>.encoding = encoding;
<span style="color: #008800; font-weight: bold;">this</span>.readerQuotas = <span style="color: #008800; font-weight: bold;">new</span> XmlDictionaryReaderQuotas();
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">MustUnderstandOffBindingElement</span>(<span style="color: #333399; font-weight: bold;">string</span> encoding, <span style="color: #333399; font-weight: bold;">string</span> mediaType)
: <span style="color: #008800; font-weight: bold;">this</span>(encoding, mediaType, MessageVersion.Soap11WSAddressing10)
{
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">MustUnderstandOffBindingElement</span>(<span style="color: #333399; font-weight: bold;">string</span> encoding)
: <span style="color: #008800; font-weight: bold;">this</span>(encoding, <span style="background-color: #fff0f0;">"text/xml"</span>)
{
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">MustUnderstandOffBindingElement</span>()
: <span style="color: #008800; font-weight: bold;">this</span>(<span style="background-color: #fff0f0;">"UTF-8"</span>)
{
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> MessageVersion MessageVersion
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.msgVersion;
}
<span style="color: #008800; font-weight: bold;">set</span>
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #008800; font-weight: bold;">value</span> == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"value"</span>);
<span style="color: #008800; font-weight: bold;">this</span>.msgVersion = <span style="color: #008800; font-weight: bold;">value</span>;
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #333399; font-weight: bold;">string</span> MediaType
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.mediaType;
}
<span style="color: #008800; font-weight: bold;">set</span>
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #008800; font-weight: bold;">value</span> == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"value"</span>);
<span style="color: #008800; font-weight: bold;">this</span>.mediaType = <span style="color: #008800; font-weight: bold;">value</span>;
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #333399; font-weight: bold;">string</span> Encoding
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.encoding;
}
<span style="color: #008800; font-weight: bold;">set</span>
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #008800; font-weight: bold;">value</span> == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"value"</span>);
<span style="color: #008800; font-weight: bold;">this</span>.encoding = <span style="color: #008800; font-weight: bold;">value</span>;
}
}
<span style="color: #888888;">// This encoder does not enforces any quotas for the unsecure messages. The </span>
<span style="color: #888888;">// quotas are enforced for the secure portions of messages when this encoder</span>
<span style="color: #888888;">// is used in a binding that is configured with security. </span>
<span style="color: #008800; font-weight: bold;">public</span> XmlDictionaryReaderQuotas ReaderQuotas
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.readerQuotas;
}
}
<span style="color: #557799;">#region IMessageEncodingBindingElement Members</span>
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> MessageEncoderFactory <span style="color: #0066bb; font-weight: bold;">CreateMessageEncoderFactory</span>()
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">MustUnderstandOffEncoderFactory</span>(<span style="color: #008800; font-weight: bold;">this</span>.MediaType,
<span style="color: #008800; font-weight: bold;">this</span>.Encoding, <span style="color: #008800; font-weight: bold;">this</span>.MessageVersion);
}
<span style="color: #557799;">#endregion</span>
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> BindingElement <span style="color: #0066bb; font-weight: bold;">Clone</span>()
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">MustUnderstandOffBindingElement</span>(<span style="color: #008800; font-weight: bold;">this</span>);
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> IChannelFactory<TChannel> BuildChannelFactory<TChannel>(BindingContext context)
{
<span style="color: #008800; font-weight: bold;">if</span> (context == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"context"</span>);
context.BindingParameters.Add(<span style="color: #008800; font-weight: bold;">this</span>);
<span style="color: #008800; font-weight: bold;">return</span> context.BuildInnerChannelFactory<TChannel>();
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> <span style="color: #333399; font-weight: bold;">bool</span> CanBuildChannelFactory<TChannel>(BindingContext context)
{
<span style="color: #008800; font-weight: bold;">if</span> (context == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"context"</span>);
<span style="color: #008800; font-weight: bold;">return</span> context.CanBuildInnerChannelFactory<TChannel>();
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context)
{
<span style="color: #008800; font-weight: bold;">if</span> (context == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"context"</span>);
context.BindingParameters.Add(<span style="color: #008800; font-weight: bold;">this</span>);
<span style="color: #008800; font-weight: bold;">return</span> context.BuildInnerChannelListener<TChannel>();
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> <span style="color: #333399; font-weight: bold;">bool</span> CanBuildChannelListener<TChannel>(BindingContext context)
{
<span style="color: #008800; font-weight: bold;">if</span> (context == <span style="color: #008800; font-weight: bold;">null</span>)
<span style="color: #008800; font-weight: bold;">throw</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">ArgumentNullException</span>(<span style="background-color: #fff0f0;">"context"</span>);
context.BindingParameters.Add(<span style="color: #008800; font-weight: bold;">this</span>);
<span style="color: #008800; font-weight: bold;">return</span> context.CanBuildInnerChannelListener<TChannel>();
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> T GetProperty<T>(BindingContext context)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #008800; font-weight: bold;">typeof</span>(T) == <span style="color: #008800; font-weight: bold;">typeof</span>(XmlDictionaryReaderQuotas))
{
<span style="color: #008800; font-weight: bold;">return</span> (T)(<span style="color: #333399; font-weight: bold;">object</span>)<span style="color: #008800; font-weight: bold;">this</span>.readerQuotas;
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">base</span>.GetProperty<T>(context);
}
}
<span style="color: #557799;">#region IWsdlExportExtension Members</span>
<span style="color: #008800; font-weight: bold;">void</span> IWsdlExportExtension.ExportContract(WsdlExporter exporter, WsdlContractConversionContext context)
{
}
<span style="color: #008800; font-weight: bold;">void</span> IWsdlExportExtension.ExportEndpoint(WsdlExporter exporter, WsdlContractConversionContext context)
{
<span style="color: #888888;">// The MessageEncodingBindingElement is responsible for ensuring that the WSDL has the correct</span>
<span style="color: #888888;">// SOAP version. We can delegate to the WCF implementation of TextMessageEncodingBindingElement for this.</span>
TextMessageEncodingBindingElement mebe = <span style="color: #008800; font-weight: bold;">new</span> TextMessageEncodingBindingElement();
mebe.MessageVersion = <span style="color: #008800; font-weight: bold;">this</span>.msgVersion;
((IWsdlExportExtension)mebe).ExportEndpoint(exporter, context);
}
<span style="color: #557799;">#endregion</span>
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">MustUnderstandOffEncoderFactory</span> : MessageEncoderFactory
{
<span style="color: #008800; font-weight: bold;">private</span> MessageEncoder encoder;
<span style="color: #008800; font-weight: bold;">private</span> MessageVersion version;
<span style="color: #008800; font-weight: bold;">private</span> <span style="color: #333399; font-weight: bold;">string</span> mediaType;
<span style="color: #008800; font-weight: bold;">private</span> <span style="color: #333399; font-weight: bold;">string</span> charSet;
<span style="color: #008800; font-weight: bold;">internal</span> <span style="color: #0066bb; font-weight: bold;">MustUnderstandOffEncoderFactory</span>(<span style="color: #333399; font-weight: bold;">string</span> mediaType, <span style="color: #333399; font-weight: bold;">string</span> charSet, MessageVersion version)
{
<span style="color: #008800; font-weight: bold;">this</span>.version = version;
<span style="color: #008800; font-weight: bold;">this</span>.mediaType = mediaType;
<span style="color: #008800; font-weight: bold;">this</span>.charSet = charSet;
<span style="color: #008800; font-weight: bold;">this</span>.encoder = <span style="color: #008800; font-weight: bold;">new</span> MustUnderstandOffEncoder(<span style="color: #008800; font-weight: bold;">this</span>);
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> MessageEncoder Encoder
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.encoder;
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> MessageVersion MessageVersion
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.version;
}
}
<span style="color: #008800; font-weight: bold;">internal</span> <span style="color: #333399; font-weight: bold;">string</span> MediaType
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.mediaType;
}
}
<span style="color: #008800; font-weight: bold;">internal</span> <span style="color: #333399; font-weight: bold;">string</span> CharSet
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.charSet;
}
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">MustUnderstandOffEncoder</span> : MessageEncoder
{
<span style="color: #008800; font-weight: bold;">private</span> XmlWriterSettings writerSettings;
<span style="color: #008800; font-weight: bold;">private</span> <span style="color: #333399; font-weight: bold;">string</span> contentType;
<span style="color: #008800; font-weight: bold;">private</span> <span style="color: #333399; font-weight: bold;">string</span> _MediaType;
<span style="color: #008800; font-weight: bold;">private</span> MessageVersion _MessageVersion;
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">MustUnderstandOffEncoder</span>(MustUnderstandOffEncoderFactory factory)
{
_MediaType = factory.MediaType;
_MessageVersion = factory.MessageVersion;
<span style="color: #008800; font-weight: bold;">this</span>.writerSettings = <span style="color: #008800; font-weight: bold;">new</span> XmlWriterSettings();
<span style="color: #008800; font-weight: bold;">this</span>.writerSettings.Encoding = Encoding.GetEncoding(factory.CharSet);
<span style="color: #888888;">//this.writerSettings.ConformanceLevel = ConformanceLevel.Fragment;</span>
<span style="color: #888888;">//this.writerSettings.OmitXmlDeclaration = true;</span>
<span style="color: #008800; font-weight: bold;">this</span>.contentType = <span style="color: #333399; font-weight: bold;">string</span>.Format(<span style="background-color: #fff0f0;">"{0}; charset={1}"</span>,
_MediaType, <span style="color: #008800; font-weight: bold;">this</span>.writerSettings.Encoding.HeaderName);
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> <span style="color: #333399; font-weight: bold;">string</span> ContentType
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">this</span>.contentType;
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> <span style="color: #333399; font-weight: bold;">string</span> MediaType
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> _MediaType;
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> MessageVersion MessageVersion
{
<span style="color: #008800; font-weight: bold;">get</span>
{
<span style="color: #008800; font-weight: bold;">return</span> _MessageVersion;
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> <span style="color: #333399; font-weight: bold;">bool</span> <span style="color: #0066bb; font-weight: bold;">IsContentTypeSupported</span>(<span style="color: #333399; font-weight: bold;">string</span> contentType)
{
<span style="color: #008800; font-weight: bold;">if</span> (<span style="color: #008800; font-weight: bold;">base</span>.IsContentTypeSupported(contentType))
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">true</span>;
}
<span style="color: #008800; font-weight: bold;">if</span> (contentType.Length == <span style="color: #008800; font-weight: bold;">this</span>.MediaType.Length)
{
<span style="color: #008800; font-weight: bold;">return</span> contentType.Equals(<span style="color: #008800; font-weight: bold;">this</span>.MediaType, StringComparison.OrdinalIgnoreCase);
}
<span style="color: #008800; font-weight: bold;">else</span>
{
<span style="color: #008800; font-weight: bold;">if</span> (contentType.StartsWith(<span style="color: #008800; font-weight: bold;">this</span>.MediaType, StringComparison.OrdinalIgnoreCase)
&& (contentType[<span style="color: #008800; font-weight: bold;">this</span>.MediaType.Length] == <span style="color: #0044dd;">';'</span>))
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">true</span>;
}
}
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">false</span>;
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> Message <span style="color: #0066bb; font-weight: bold;">ReadMessage</span>(ArraySegment<<span style="color: #333399; font-weight: bold;">byte</span>> buffer, BufferManager bufferManager, <span style="color: #333399; font-weight: bold;">string</span> contentType)
{
<span style="color: #333399; font-weight: bold;">byte</span>[] msgContents = <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #333399; font-weight: bold;">byte</span>[buffer.Count];
Array.Copy(buffer.Array, buffer.Offset, msgContents, <span style="color: #6600ee; font-weight: bold;">0</span>, msgContents.Length);
bufferManager.ReturnBuffer(buffer.Array);
MemoryStream stream = <span style="color: #008800; font-weight: bold;">new</span> MemoryStream(msgContents);
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #0066bb; font-weight: bold;">ReadMessage</span>(stream, <span style="color: #333399; font-weight: bold;">int</span>.MaxValue);
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> Message <span style="color: #0066bb; font-weight: bold;">ReadMessage</span>(Stream stream, <span style="color: #333399; font-weight: bold;">int</span> maxSizeOfHeaders, <span style="color: #333399; font-weight: bold;">string</span> contentType)
{
XmlReader reader = XmlReader.Create(stream);
<span style="color: #008800; font-weight: bold;">return</span> Message.CreateMessage(reader, maxSizeOfHeaders, <span style="color: #008800; font-weight: bold;">this</span>.MessageVersion);
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> ArraySegment<<span style="color: #333399; font-weight: bold;">byte</span>> WriteMessage(Message message, <span style="color: #333399; font-weight: bold;">int</span> maxMessageSize, BufferManager bufferManager, <span style="color: #333399; font-weight: bold;">int</span> messageOffset)
{
MemoryStream stream = <span style="color: #008800; font-weight: bold;">new</span> MemoryStream();
XmlWriter writer = XmlWriter.Create(stream, <span style="color: #008800; font-weight: bold;">this</span>.writerSettings);
message.WriteMessage(writer);
writer.Close();
<span style="color: #333399; font-weight: bold;">string</span> decoded = Encoding.UTF8.GetString(stream.ToArray());
decoded = decoded.Replace(<span style="background-color: #fff0f0;">"mustUnderstand=\"1\""</span>, <span style="background-color: #fff0f0;">"mustUnderstand=\"0\""</span>);
stream.Close();
<span style="color: #333399; font-weight: bold;">byte</span>[] messageBytes = System.Text.Encoding.UTF8.GetBytes(decoded);
<span style="color: #333399; font-weight: bold;">int</span> messageLength = messageBytes.Length;
<span style="color: #888888;">//byte[] messageBytes = stream.GetBuffer();</span>
<span style="color: #888888;">//int messageLength = (int)stream.Position;</span>
<span style="color: #888888;">//stream.Close();</span>
<span style="color: #333399; font-weight: bold;">int</span> totalLength = messageLength + messageOffset;
<span style="color: #333399; font-weight: bold;">byte</span>[] totalBytes = bufferManager.TakeBuffer(totalLength);
Array.Copy(messageBytes, <span style="color: #6600ee; font-weight: bold;">0</span>, totalBytes, messageOffset, messageLength);
ArraySegment<<span style="color: #333399; font-weight: bold;">byte</span>> byteArray = <span style="color: #008800; font-weight: bold;">new</span> ArraySegment<<span style="color: #333399; font-weight: bold;">byte</span>>(totalBytes, messageOffset, messageLength);
<span style="color: #008800; font-weight: bold;">return</span> byteArray;
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> <span style="color: #008800; font-weight: bold;">void</span> <span style="color: #0066bb; font-weight: bold;">WriteMessage</span>(Message message, Stream stream)
{
XmlWriter writer = XmlWriter.Create(stream, <span style="color: #008800; font-weight: bold;">this</span>.writerSettings);
message.WriteMessage(writer);
writer.Close();
}
}
}
</pre>
</div>
<br />
While that is all the code I ended up using. There is one more class I wanted to share for completeness. During my research, at one point I thought my problem was stemming from passing in username and password security credentials in the header without a nonce and time stamp.<br />
<br />
This is a feature Microsoft apparently purposefully excluded because they felt it was less secure and should not be promoted. Fortunately I ended up not needing it, but I feel for those who do need it and must deal with Microsoft taking the ability away from them because they think they know better than the developer.<br />
<br />
Here is the class I ended up using successfully to fix this particular issue. I copied it almost verbatim from <a href="https://weblog.west-wind.com/posts/2012/Nov/24/WCF-WSSecurity-and-WSE-Nonce-Authentication">Rick Strahl's Blog</a>.<br />
<br />
<!-- HTML generated using hilite.me --><b>Custom Credentials Classes</b><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Collections.Generic</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Linq</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Text</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Threading.Tasks</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Description</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.ServiceModel.Security</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.IdentityModel.Tokens</span>;
<span style="color: #008800; font-weight: bold;">using</span> <span style="color: #0e84b5; font-weight: bold;">System.Security.Cryptography</span>;
<span style="color: #008800; font-weight: bold;">namespace</span> <span style="color: #0e84b5; font-weight: bold;">Services</span>
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">WsNonceCustomCredentials</span> : ClientCredentials
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">WsNonceCustomCredentials</span>() { }
<span style="color: #008800; font-weight: bold;">protected</span> <span style="color: #0066bb; font-weight: bold;">WsNonceCustomCredentials</span>(WsNonceCustomCredentials cc) : <span style="color: #008800; font-weight: bold;">base</span>(cc) { }
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> System.IdentityModel.Selectors.SecurityTokenManager CreateSecurityTokenManager()
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">WsNonceCustomSecurityTokenManager</span>(<span style="color: #008800; font-weight: bold;">this</span>);
}
<span style="color: #008800; font-weight: bold;">protected</span> <span style="color: #008800; font-weight: bold;">override</span> ClientCredentials <span style="color: #0066bb; font-weight: bold;">CloneCore</span>()
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">WsNonceCustomCredentials</span>(<span style="color: #008800; font-weight: bold;">this</span>);
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">WsNonceCustomSecurityTokenManager</span> : ClientCredentialsSecurityTokenManager
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">WsNonceCustomSecurityTokenManager</span>(WsNonceCustomCredentials cred) : <span style="color: #008800; font-weight: bold;">base</span>(cred) { }
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">override</span> System.IdentityModel.Selectors.SecurityTokenSerializer CreateSecurityTokenSerializer(System.IdentityModel.Selectors.SecurityTokenVersion version)
{
<span style="color: #008800; font-weight: bold;">return</span> <span style="color: #008800; font-weight: bold;">new</span> <span style="color: #0066bb; font-weight: bold;">WsNonceCustomTokenSerializer</span>(System.ServiceModel.Security.SecurityVersion.WSSecurity11);
}
}
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">WsNonceCustomTokenSerializer</span> : WSSecurityTokenSerializer
{
<span style="color: #008800; font-weight: bold;">public</span> <span style="color: #0066bb; font-weight: bold;">WsNonceCustomTokenSerializer</span>(SecurityVersion sv) : <span style="color: #008800; font-weight: bold;">base</span>(sv) { }
<span style="color: #008800; font-weight: bold;">protected</span> <span style="color: #008800; font-weight: bold;">override</span> <span style="color: #008800; font-weight: bold;">void</span> <span style="color: #0066bb; font-weight: bold;">WriteTokenCore</span>(System.Xml.XmlWriter writer,
System.IdentityModel.Tokens.SecurityToken token)
{
UserNameSecurityToken userToken = token <span style="color: #008800; font-weight: bold;">as</span> UserNameSecurityToken;
<span style="color: #008800; font-weight: bold;">if</span> (userToken == <span style="color: #008800; font-weight: bold;">null</span>) <span style="color: #008800; font-weight: bold;">return</span>;
<span style="color: #333399; font-weight: bold;">string</span> tokennamespace = <span style="background-color: #fff0f0;">"o"</span>;
DateTime created = DateTime.Now;
<span style="color: #333399; font-weight: bold;">string</span> createdStr = created.ToString(<span style="background-color: #fff0f0;">"yyyy-MM-ddThh:mm:ss.fffZ"</span>);
<span style="color: #888888;">// unique Nonce value - encode with SHA-1 for 'randomness'</span>
<span style="color: #888888;">// in theory the nonce could just be the GUID by itself</span>
<span style="color: #333399; font-weight: bold;">string</span> phrase = Guid.NewGuid().ToString();
<span style="color: #333399; font-weight: bold;">var</span> nonce = GetSHA1String(phrase);
<span style="color: #888888;">// in this case password is plain text</span>
<span style="color: #888888;">// for digest mode password needs to be encoded as:</span>
<span style="color: #888888;">// PasswordAsDigest = Base64(SHA-1(Nonce + Created + Password))</span>
<span style="color: #888888;">// and profile needs to change to</span>
<span style="color: #888888;">//string password = GetSHA1String(nonce + createdStr + userToken.Password);</span>
<span style="color: #333399; font-weight: bold;">string</span> password = userToken.Password;
writer.WriteRaw(<span style="color: #333399; font-weight: bold;">string</span>.Format(
<span style="background-color: #fff0f0;">"<{0}:UsernameToken u:Id=\""</span> + token.Id +
<span style="background-color: #fff0f0;">"\" xmlns:u=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\">"</span> +
<span style="background-color: #fff0f0;">"<{0}:Username>"</span> + userToken.UserName + <span style="background-color: #fff0f0;">"</{0}:Username>"</span> +
<span style="background-color: #fff0f0;">"<{0}:Password Type=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText\">"</span> +
password + <span style="background-color: #fff0f0;">"</{0}:Password>"</span> +
<span style="background-color: #fff0f0;">"<{0}:Nonce EncodingType=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary\">"</span> +
nonce + <span style="background-color: #fff0f0;">"</{0}:Nonce>"</span> +
<span style="background-color: #fff0f0;">"<u:Created>"</span> + createdStr + <span style="background-color: #fff0f0;">"</u:Created></{0}:UsernameToken>"</span>, tokennamespace));
}
<span style="color: #008800; font-weight: bold;">protected</span> <span style="color: #333399; font-weight: bold;">string</span> <span style="color: #0066bb; font-weight: bold;">GetSHA1String</span>(<span style="color: #333399; font-weight: bold;">string</span> phrase)
{
SHA1CryptoServiceProvider sha1Hasher = <span style="color: #008800; font-weight: bold;">new</span> SHA1CryptoServiceProvider();
<span style="color: #333399; font-weight: bold;">byte</span>[] hashedDataBytes = sha1Hasher.ComputeHash(Encoding.UTF8.GetBytes(phrase));
<span style="color: #008800; font-weight: bold;">return</span> Convert.ToBase64String(hashedDataBytes);
}
}
}
</pre>
</div>
<br />
My final assessment of the situation is that Microsoft really screwed things up by removing this feature from WCF. However, with a lot of work, it is possible to hack the change back in. Hopefully Microsoft will fix this problem some day. Whether it is an oversight or a feature on their part, the decision they made was a bad one for those of us actually having to do the work to undo what they did.</div>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com9tag:blogger.com,1999:blog-703101161281072462.post-48633539875948845282017-10-12T07:28:00.000-07:002017-10-12T07:28:41.939-07:00NetTeirs and ReflectionBy now <a href="https://github.com/netTiers/netTiers">NetTiers</a> is a rather aging templating system. Many people have complained that it is not being actively maintained and that there are a lot of errors in it, and to some degree they are correct.<br />
<br />
NetTiers gets random updates from those using it, but not very frequently. The code is definitely not bug free, however it is not chalk full of bugs either. It does rely on Microsoft's EnterpriseLibrary sweet, which is no longer maintained; although the documentation claims this is not a requirement.<br />
<br />
But NetTiers has several key features and functionality that had me pick it up after testing multiple data access layer solutions.<br />
<br />
<ul>
<li>It generates a direct copy of your database into C# code with just the click of a button. Very few other data access layers can claim that, and their solutions do not feel as smooth as the Codesmith platform that NetTiers runs on.</li>
<li> During the generation process NetTiers standardizes the table and column names found in your database, so they all start capitalized, words it identifies are capitalized, and table prefixes can be optionally stripped off.</li>
<li>It uses meta data from sql regarding indexes and foreign keys to auto generate methods to get data out of the database.</li>
<li>That meta data also allows links to be made between objects in the generated c# code so you can load and reference foreign key relationships easily. </li>
<li>Stored Procedures can be auto generated for those who want the added security and to avoid dynamic sql.</li>
<li>It has a reasonably powerful parameterized sql engine when required.</li>
<li>A caching layer allows the developer to cash most result sets that come back from the database without additional work. Which can be useful for small static tables.</li>
<li>It picks up custom stored procedures and adds methods onto the generated objects to allow the developer to call them without any additional effort. </li>
<li>There are quite a few template options to allow the developer to customize the resulting code to their environment.</li>
</ul>
Many data access layers claim features similar to these, although oftentimes with less control over them. However, there are three features that make NetTiers unique and keep it valuable and relevant:<br />
<br />
The direct copy of the database into standard c# code takes a huge load off of the developer who no longer has to write all that plumbing code. The generated code is in the form of normal C# objects that match the names in your database. This means there is no guesswork as to what an object name is going to be. It also means full object oriented and intellisense capable access to every object in the database.<br />
<br />
Generated code is compile time safe; so most errors in the database will show up right away and not allow the code to compile. This code has been used by many people lots of times, so most major bugs have been worked out already. Having it all generated the same way every time nearly eliminates the ability for a random error to sneak into the data access layer.<br />
<br />
One of my favorite features is actually a byproduct of having the data access layer so standardized. The ability to use reflection heavily. Because there are so many standards built into nettiers, you can always find the correct get or save methods or primary keys regardless of the object you are working with. The generated code in these areas all follows the same pattern.<br />
<br />
While not perfect ( I have submitted a few corrections back to the code base myself ), the ability to auto generate an entire DAL without effort, and the ability to rely heavily on reflection when using these objects, has made me a believer in NetTiers for many of the projects I work on.<br />
<br />
It is not the only solution available. But for server based C# projects it definitely does shine as a way to drastically reduce development time.<br />
<br />
<br />
<br />Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-67741036962713726572017-05-16T14:14:00.002-07:002017-06-11T12:03:13.553-07:00WanCam wireless camera HW0036I made a mistake in getting this camera. Despite it having the features it advertised, this camera is missing an RJ45 jack. For security reasons I never buy a camera without a wired connection.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-0xmIej_whkA/WRtqU3fTWoI/AAAAAAAA77k/xBVJaQeaQRUqGaIUxpi0WB02PUgU1LT6QCKgB/s1600/20170516_140353.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://2.bp.blogspot.com/-0xmIej_whkA/WRtqU3fTWoI/AAAAAAAA77k/xBVJaQeaQRUqGaIUxpi0WB02PUgU1LT6QCKgB/s320/20170516_140353.jpg" width="240" /></a></div>
<br />
<br />
This camera does work correctly, however they made a couple of annoying design decisions when making this model. First, the http port by default is 81 rather than the standard 80. Second they also deviated from the standard OnVif ports and choose to use 10080 for this service; and it is not configurable.<br />
<br />
It does not support https, which I consider a negative, although not horrible as I rarely use the web interface once configured.<br />
<br />
Fortunately you can change the http port, and you can change the default password. The camera works very well with Milestone. As long as you only want wireless then this is a good camera, however because of that reason I would never recommend, or purchase another one.<br />
<br />
EDIT: Within a week of owning this camera it began to have issues turning on. A few days later the problem was so bad the camera no longer functioned. It is possible that I simply got a lemon, or this particular model has a bad design flaw. I was able to get a partial refund, threw the camera away, and will never get another of this model.Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-23605796751005745492017-05-15T20:55:00.002-07:002017-05-16T07:53:00.682-07:00KanKun Wireless PlugI recently purchased a wireless KanKun plug, specifically because I heard that it was running a version of OpenWRT and could easily be modified.<br />
<br />
I found the primary <a href="http://www.homeautomationforgeeks.com/openhab_http.shtml">directions here</a>.<br />
<br />
There were a couple of challenges. The first was getting the plug on the wireless network. The Chinese directions and app did not function at all, and I tried them multiple times.<br />
<br />
The good news is that the plug successfully resets all failed attempts very well with its reset process. Eventually I was successful following these steps:<br />
<br />
- Use a laptop to connect to the wireless network the plug defaults to creating.<br />
- Lookup the Plugs IP based on the DHCP address assigned to the laptop because it did not create the network that most examples were claiming it would.<br />
- Use Putty to SSH into the plug with a username of 'root'. My default password was p9z34c, however others have claimed it might be admin or 1234.<br />
- Edit the following file with 'vi': /etc/config/wireless<br />
<br />
config wifi-device 'radio0'<br />
option type 'mac80211'<br />
option channel '9'<br />
option hwmode '11ng'<br />
option path 'platform/ar933x_wmac'<br />
list ht_capab 'SHORT-GI-20'<br />
list ht_capab 'SHORT-GI-40'<br />
list ht_capab 'RX-STBC1'<br />
list ht_capab 'DSSS_CCK-40'<br />
option htmode 'HT20'<br />
option disabled '0'<br />
option country 'CN'<br />
<br />
config wifi-iface<br />
option device 'radio0'<br />
option network 'wwan'<br />
option ssid 'MyWirelessSID'<br />
option encryption 'psk2'<br />
option mode 'sta'<br />
option key 'MyWirelessPassword'<br />
<div>
<br /></div>
<div>
I made sure the channel was set correctly since I run my wireless network on 9 rather than the standard 11. I also struggled a bit with the encryption since I run WPA2 Personal; I finally got the very logical instructions to set encryption to 'psk2'.</div>
<div>
<br /></div>
<div>
- The second critical file to edit is: /etc/config/network</div>
<div>
<div>
<br /></div>
<div>
config interface 'loopback'</div>
<div>
option ifname 'lo'</div>
<div>
option proto 'static'</div>
<div>
option ipaddr '127.0.0.1'</div>
<div>
option netmask '255.0.0.0'</div>
<div>
<br /></div>
<div>
config globals 'globals'</div>
<div>
option ula_prefix 'fd59:45ed:8ead:0000:/48'</div>
<div>
<br /></div>
<div>
config interface 'lan'</div>
<div>
option ifname 'eth0'</div>
<div>
option force_link '1'</div>
<div>
option type 'bridge'</div>
<div>
option proto 'static'</div>
<div>
option ipaddr '192.168.145.253'</div>
<div>
option netmask '255.255.255.0'</div>
<div>
option ip6assign '60'</div>
<div>
<br /></div>
<div>
config interface 'wan'</div>
<div>
option proto 'dhcp'</div>
<div>
option ifname 'eth1'</div>
<div>
<br /></div>
<div>
config interface 'wwan'</div>
<div>
option proto 'static'</div>
<div>
option ipaddr '192.168.1.32'</div>
<div>
option gateway '192.168.1.1'</div>
<div>
option netmask '255.255.255.0'</div>
</div>
<div>
<br /></div>
<div>
I modified the MAC address in this file, adding the '0000', however leaving it empty would probably have been fine as well from what others have said; and considering it was working prior. Adding the last 'config interface wwan' section is the most critical. It is used by the wireless file for interface settings and specifies the static IP network necessary to communicate on this network.</div>
<div>
<br /></div>
<div>
- 'reboot' the plug</div>
<div>
- With the plug now successfully connecting to my network I was able to return to my main computer for further configuration.</div>
<div>
- My next step was to copy in a simple CGI script someone created to allow me to remotely control the plug. I did have to modify the following script slightly because my RELAY_CTRL was different from the ones they were using.</div>
<div>
- I had to create the folder /www/cgi-bin</div>
<div>
- And create the following file: /www/cgi-bin/relay.cgi</div>
<div>
<div>
<br /></div>
<div>
#!/bin/sh</div>
<div>
echo "Content-Type: text/plain"</div>
<div>
echo "Cache-Control: no-cache, must-revalidate"</div>
<div>
echo "Expires: Sat, 26 Jul 1997 05:00:00 GMT"</div>
<div>
echo</div>
<div>
<br /></div>
<div>
RELAY_CTRL=/sys/class/leds/i-konke\:red\:relay/brightness</div>
<div>
<br /></div>
<div>
case "$QUERY_STRING" in</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>state) </div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>case "`cat $RELAY_CTRL`" in</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>0) echo "OFF"</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>;;</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>1) echo "ON"</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>;;</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>esac</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>;;</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>on) </div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>echo 1 > $RELAY_CTRL</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>echo OK</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>;;</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>off) </div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>echo 0 > $RELAY_CTRL</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>echo OK</div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span>;;</div>
<div>
esac</div>
</div>
<div>
<br /></div>
<div>
- Unfortunately I discovered that the uhttpd service did not exist on my plug. I had to turn to a very helpful Internet community who was able to give me the required files. You can <a href="https://drive.google.com/open?id=0Bw4h_PZMcXOGNzQ1cDA2VGVtS0U">download them here</a>.</div>
<div>
- Inside the rar file you should see a kkplug folder with the most likely folder structure and the files that go in each folder. Use WinSCP to copy the files to the plug.<br />
- Run the following command</div>
<div>
/etc/init.d/uhttpd enable</div>
<div>
- This creates a sym link that causes the plug to execute the startup file on boot.</div>
<div>
- 'reboot' again to test and verify that the service will start correctly.</div>
<div>
<br /></div>
<div>
- When the plug comes up you should be able to use a web browser and control the plug in this way.</div>
<div>
http://192.168.1.32/cgi-bin/relay.cgi?on</div>
<div>
<br /></div>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-59108608964234986142017-03-12T13:55:00.001-07:002017-03-23T07:48:24.108-07:00Knewmart Indoor IP CameraI am always on the lookout for inexpensive yet quality cameras to integrate with my security system. If you buy a camera today from almost any store it is nearly a given that it will come with an app that you can use to view the camera remotely. But aside from a few small motion detection perks that is usually about the end of it. Many cameras do not play well with other cameras, or make it easy and cheap to actually record like a real security system.<br />
<br />
For this reason I have come up with the following rules when buying cameras:<br />
1. It must support OnVif. This is a standards protocol that guarantees the camera will at least try and play nice with most of the security systems out there.<br />
2. It must support H.264 encoding. This is a compression algorithm that significantly reduces the amount of storage required when recording for long periods.<br />
3. It must have an Ethernet jack. Many people like the idea of going wireless for convenience, but even wireless cameras need a power cord. However, wireless has the major downside of being insecure.<br />
- Not only is it much easier to hack into a wireless feed.<br />
- But if you put the camera next to your microwave and turn it on you will realize it is a piece of cake to simply overwhelm the camera with radiation, disabling the feed.<br />
- Also a nice quality camera will be sending so much data over the wireless network it can overwhelm a normal wireless router, effectively rendering your wireless network useless.<br />
- And even if you decided to give it its own wireless network, you are still cluttering the radio waves which could end up causing various problems for you in the future.<br />
4. It really should allow for a Static IP to be set, it is just easier to manage networks that way.<br />
5. If it is an outdoor camera it must support POE. Power Over Ethernet allows me to run just one low voltage cable to the camera. Trying to mess with high voltage extension cords and a power cable just is not worth the trouble.<br />
<br />
As a Note. For outdoor cheap cameras I have had better luck with the few large LED configuration over the numerous small LEDs for night vision. Cheaper cameras are not made with the same care, and you can end up with the small LEDs not being positioned correctly and washing out your image at night.<br />
<br />
Camera technology is moving fast, but after testing quite a few brands I had settled on a really nice outdoor camera that met all my requirements and worked very well indoors also. But then two new brands came on the scene, Wancam and Knewmart.<br />
<br />
Today I am looking at the Knewmart (no model).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-okKcpmYwRhM/WMYdMlpglWI/AAAAAAAA5mY/iRMW_Oh2JTkS9bSkTBzxzfo05zsWUvJiwCKgB/s1600/20170312_211117.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="https://1.bp.blogspot.com/-okKcpmYwRhM/WMYdMlpglWI/AAAAAAAA5mY/iRMW_Oh2JTkS9bSkTBzxzfo05zsWUvJiwCKgB/s320/20170312_211117.jpg" width="320" /></a></div>
<br />
<br />
Positives<br />
- It fits all my requirements for an indoor camera (eg, it supports OnVif and H.264 and allows for a static IP to be set, and has an Ethernet jack)<br />
- It has wireless built in in case I do ever need it, and the ability to disable it for security if I don't use it.<br />
- For those of us who do not like installing more apps on our phone it has a built in web server so I can configure it from my computer, which I love. It is the first web interface I have seen on a less than $50 camera that actually works well.<br />
- The web configuration interface is very simple and easy to use and really started me falling in love with this camera immediately.<br />
- The camera ships with DHCP enabled by default, so it auto connects to your wired network when you plug it in. This is a much better design in my opinion than cameras that try and stand up their own wireless network for configuration.<br />
- It has a default username/password of admin/admin. It does have a user and guest account, which is nice, but it has no way to disable them, so make sure you change all the passwords.<br />
- For those who are paranoid about configuring their own cameras, it does come with a QR code to auto configure basic connectivity in their native app.<br />
- It has the ability for dual streams at different quality levels, which was unexpected although it is probably becoming more standard in the industry.<br />
- For those who do not run separate NVR software, it has the built in ability to trigger alarms during a time span you schedule and send you an email with a picture of what caused the alarm. However, I did not test this.<br />
- It has little motors so you can pan and tilt it, which is new for the less expensive cameras, and I absolutely love it. This Knewmart has the widest range of motion I have experienced yet, it <span style="background-color: white; color: #111111; font-family: "arial" , sans-serif; font-size: 13px;">Pan:</span> 355°, Tilt: 90°<br />
- It does work with the E-View7 app, which I liked because I already use that app for other cameras. They also made a special P2PIPC app according to the directions that came with it.<br />
- For those who are not quite ready to built an NVR system, but still want to record some video locally, it does come with a MicroSD card slot. While this does seem to be a popular feature, it is a feature I have never personally found a good use for.<br />
- It also comes with the ability to talk out of it and hear through it, another set of features I absolutely love in this camera.<br />
- There is port labeled as a headphone jack, although I think the picture is misleading, I believe it is a Microphone jack in case you want better audio reception. The OnVif api supports this conclusion by listing two microphones.<br />
- I use Milestone's XProtect software, and after enabling OnVif it had no issue connecting to the camera on the first try. Microphone and pan/tilt work great.<br />
- As far as cameras go, this thing actually looks very nice. I question whether it is a bit too big and bulky for an indoor camera that you typically want to be unobtrusive; but the smooth fluids lines on it help make it a more attractive decoration. Plus there is the argument that being bigger means better quality.<br />
- It has the built-in ability to contact time.windows.com to set it's own time and handle daylight savings time; you just have to choose the right time zone. This is a very handy feature that is often overlooked.<br />
- It has the standard Flip and Mirror options for the picture, which are really nice when you want to mount the camera in an odd place, like the ceiling.<br />
- I will give this company credit for effort in support also, this is a nice in-expensive camera, and they respond to support requests. It may not be American level support, but they also do not leave you hanging which is really nice.<br />
<div>
- It does have a reset button, I mention this obvious feature only because I have actually seen cameras without one.<br />
- It seamlessly switches between wireless and wired without requiring a hard reboot.</div>
<div>
- And finally it takes a unique approach to wall and ceiling mounting. Most cameras like this have holes in their bottom you just slide over screws. This camera ships with a pretty configurable mounting arm. While I am not sure the aesthetics of it are great, it certainly does allow for extreme flexibility in how and where you mount the camera.</div>
<div>
<br /></div>
Negatives<br />
- The directions that came with the camera are very basic and intended for a user who only cares about getting the camera running. If you want to do anything advanced with it you will probably need to use the web interface on your computer and do a little techie research if you do not already understand some of the terminology.<br />
- It does have a small glitch/feature where you have to manually unplug it after making some change to the OnVif settings to actually make OnVif work, it comes enabled by default it just does not work by default. I have experienced this with the wancam as well.<br />
<div>
- I was disappointed that the speaker does not work over OnVif with my XProtect NVR system, but I expected that from other similar cameras I have had. Hopefully some day they add that feature.<br />
- It does not support POE. But for an indoor camera that is not a deal breaker for me.</div>
<div>
- Like other cameras of this type, when you first turn it on it tests its range of motion, likely to configure the limits in the software. It has always concerned me that the camera trying to go past its limit during these tests will eventually burn out the motor, but so far that has not happened.<br />
- It does not support Https, however for those who are that paranoid about security (like me) they are probably running their cameras on a closed internal secure network anyway, so it would not be much of an issue.<br />
- If you tilt the camera too far down then its night vision LEDs reflect on itself causing the image to blur white.</div>
<div>
- The default IRCut value seems to be too low, in low light the camera starts flickering the IR on and off. You can either fiddle with the settings, or just switch it to manual mode.<br />
- It is not very easy to unplug a network cable once plugged into it, especially with bigger fingers.<br />
- When used via OnVif with the XProtect software, there is an error in the OnVif protocol when used over wireless that causes a lag in the image frames triggering XProtect to constantly report it as disconnected. Oddly enough, you can get around this error by plugging it into a network cable, waiting for the transition, then unplugging the network cable. It will then work correctly for awhile.</div>
<br />
All in all, this is a great camera that can support a range of users from those starting out, to those who are quite a ways along in building a system. If it were not for the larger size (roughly double it's WanCam counterpart) it would become my go-to indoor camera for home systems. However, it beats out other inexpensive cameras in so many areas that it is still going to be at the top of my list for consideration when I recommend cameras to people<br />
<br />Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-76900423684202452262016-11-22T08:00:00.002-08:002016-11-22T08:01:40.745-08:00NetTiers Ambiguous match foundI recently ran into a bug that took me longer than it should have to figure out.<br />
<br />
I was migrating an old code base to CodeSmith's NetTiers and started getting the error "Ambiguous match found". Results from the Internet all pointed the finger at Microsoft's .net, albeit earlier versions than version 4.5 that I was using.<br />
<br />
While I did become convinced that it was a Microsoft issue relating to two object attributes with the same name just different letter casing, that did not solve my issue for me because my code was all being auto generated by NetTiers. NetTiers was using reflection to get some attribute values, so I figured one of my table or column names was conflicting with some internal variable that NetTiers was using.<br />
<br />
The solution turned out to be rather simple. I ran the code in debug mode, triggered the crash, and started inspecting the variables in play. Particularly the one NetTiers was trying to find using reflection. It turned out that NetTiers was looking for an attribute named "Item", which corresponded to a table column of the same name, and was blowing up in the attempt. I renamed that column in my database, regenerated the code, and the problem was solved.Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-33342458482982928502016-10-19T09:51:00.001-07:002018-11-05T13:07:21.625-08:00Downloading Ancestry.com Media<a href="http://ancestry.com/">Ancestry.com</a> has never created a good system to back up the data that is on their site. The only option people have is to download the GEDCOM file, which is often a mere fraction of the data they have invested in the site.<br />
<div>
<br /></div>
<div>
For years now my family has been investing time in building a well documented family tree on <a href="http://ancestry.com/">ancestry.com</a>, and I have made sure to frequently backup the GEDCOM file so we didn't lose the data. With thousands of media files on the site I have become increasingly concerned about the loss of a major portion of our research if I was unable to download them.</div>
<div>
<br /></div>
<div>
To combat this problem I finally wrote a <a href="https://chrome.google.com/webstore/detail/ancestry-media-download/ohkkponpfoijonbbedcehkhfejlobmkb?utm_source=chrome-ntp-icon">small Chrome extension</a> that allows you to screen scrape your media files in an automated fashion. It is not perfect, but considering it is the only way to get your data, it is nice to have.</div>
<div>
<br /></div>
<div>
The extension downloads the files in mass and dumps them all into your default downloads folder. So, you should clear all files out of your default downloads folder before using this extension so you know that all downloaded files belong to Ancestry.</div>
<div>
<br /></div>
<div>
After installing this extension, you need to:</div>
<div>
- log into your ancestry.com account</div>
<div>
- open up your tree</div>
<div>
- open up the Media page, which is one of the items on the menu inside the tree</div>
<div>
- click on either the Photos or Stories tab, currently the All, Audio, and Video tabs are not supported</div>
<div>
- OPTIONAL: click on the page you want to resume downloading at. This is useful if you need to stop downloading for a period of time, say if you only want to run the downloads during the night. You will likely end up with a few duplicate files if you attempt this.</div>
<div>
- click on the Extension icon in your Chrome browser upper right corner, and click the Download Media button<br />
<br />
<span id="bc_0_72b+seedd1gD" kind="d">If your browser is asking you to save every file then the extension will not be able to correctly calculated the required download time and it will go on to the next page before you
are done. It is expecting the download to automatically start. To fix this problem, go
into your browser settings and adjust it so downloads do NOT always
prompt you for a save location. Instead set a default location for it
to use every time. The only way I could fix this issue would be
to create a setting that allows the user to manually click the next page
button instead of having it automated. And I think that would kind of
defeat the point of the extension.</span> </div>
<div>
<br /></div>
<div>
If the downloads are occurring too quickly and swamping your computer then let me know. I am planning on allowing the download speed to be adjusted in the future. Currently a file is downloaded every 2 seconds, which works well for me, but for people who upload lots of large files it will not work as well for.<br />
<br />
NOTE: There is a small chance that using this is against Ancestry's terms of use. Their terms of use forbid scraping tools, however the language seems directed at automated scripts that run in the background. This tool is simply a browsing aid performing clicks for you, but your browser is still open and doing all the browsing and preventing you from doing other browsing simultaneously.<br />
<br />
EDIT 10/2018:<br />
<br />
Just released a new version that works with Ancestry's latest photo hosting method. In order to get it working again the extension is going to need permissions to open and close windows as well as download files. When you first try and use the photo downloading feature Chrome will likely popup a few requests asking you to let the extension do downloads and open and close windows. So expect that as you try to get it going.<br />
<br />
This version also handles the Start/Stop downloading sensing better. Although, I have noticed that sometimes it takes one or two tries to get it to stop downloading depending on at what point in its execution you click the stop button.</div>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com155tag:blogger.com,1999:blog-703101161281072462.post-79556118958455599172016-07-20T12:42:00.000-07:002016-07-20T13:00:19.279-07:00LastPass Password ManagerMost people I have met know that long complex passwords are a good idea. But very few people know of a good way to keep track of all their passwords. I personally have well over 200 accounts that I need to track passwords for. Many people, myself included, get stuck in the habit of picking out a handful of passwords and using them for everything.<br />
<br />
Over the years I have tried a few password managers including <a href="http://keepass.info/">KeyPass </a>and <a href="https://www.passpack.com/">PassPack</a>. They all had nice features, but they all ended up being such a pain that I was never able to use them to create truly random unique passwords for every account I have. KeyPass did not sync across computers, PassPack had a very limited free version, and the issues go on.<br />
<br />
Then I discovered <a href="https://lastpass.com/">LastPass</a>. It took me a month of usage to fully start trusting it, but slowly I have started using more and more features of it. The best part is the amount of features they offer for free. And their first paid level ( that many people will want ) is only $12/yr at the moment. So price for features was exactly what I was looking for.<br />
<br />
LastPass syncs across all my computers, or phones for free (paid version can do both at the same time). It auto fills my website passwords with a Chrome plugin. It has been gamified to show me my overall security score and compare it to the score of other users. It supports lots of dual factor authentication options. It allows me to re-prompt for my password and do other security enhancements on a per password basis. It even has built in form fills, both generic and tailored to sites that require more than just a username and password to login. And critically, it allows me to backup all the data in my account.<br />
<br />
Having an easy backup system is critical for me, because no matter how good software is, and how much they claim they are not going anywhere, anything can happen. Their website could crash, they could go out of business, my encrypted data could get corrupted, I could forget my master password, etc. So I use the backup feature frequently to export a plain text copy of all my data including passwords, I then encrypt the file and store it. So if anything every happens, or a better software comes along, I can easily recover using my backup file.<br />
<br />
In addition to a good backup system, I also wanted to be sure they could not be hacked. Unfortunately they have been hacked at least twice. However, the <a href="http://arstechnica.com/security/2015/06/hack-of-cloud-based-lastpass-exposes-encrypted-master-passwords/">articles</a> and <a href="https://blog.lastpass.com/2015/06/lastpass-security-notice.html/">responses</a> to these hacks were so good that it made me feel even more secure using the software; no critical data was lost, users could easily remove the danger by changing the one piece of lost data, and that lost data was encrypted so heavily that it was unlikely anyone would figure it out anyway. I did use their advanced settings to increase my security above the default level though.<br />
<br />
A few things I did to make my account more secure:<br />
- Make sure all my client browser plugins log me off after a reasonable period of inactivity<br />
- Go into my vaults advanced settings:<br />
- - increase the client-side rounds to something greater than 50k. Due to LastPass's speed warnings I did this in increments testing the speed both on my phone and computer. I also backed up my data first in case the re-encryption corrupted everything.<br />
- - restricted logins to just my country<br />
- - added dual factor authentication<br />
- - reduced my website logoff settings down to a dayJeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-78117420372662429522016-05-05T19:52:00.001-07:002016-06-09T07:25:45.871-07:00Studying for Microsoft 70-463 ( Implementing a Data Warehouse with Microsoft SQL Server 2012 )For those of you out there who wish to get your MCSA SQL Server 2012, this is my path of choice, I hope it helps some of you out:<br />
<br />
<br />
<ol>
<li>Take the free <a href="https://mva.microsoft.com/en-US/training-courses/implementing-a-data-warehouse-with-sql-server-jump-start-8257">Jump Start course from MVA</a></li>
<li>Practice and brush up on features using the<a href="https://www.microsoft.com/en-us/server-cloud/support/learning-center/virtual-labs.aspx"> Microsoft Virtual Lab</a> for Exploring MDS and DQS, about half way down the page.</li>
<li>Pull together <a href="http://goo.gl/7Jo1Zj">60 practice question</a>s. which you can take as well.</li>
<li>Pull together 60 <a href="http://goo.gl/D3uDU7">more practice questions</a>.</li>
<li>Pull together 60<a href="http://goo.gl/rZG0hW"> more practice questions</a>.</li>
<li>Pull together a <a href="http://goo.gl/o3WbND">few more practice questions</a>.</li>
<li>Found someone else's <a href="https://www.goconqr.com/en-US/p/5656856-Microsoft-70-463-Practice-Questions-4-quizzes">86 practice questions</a>.</li>
<li>Buy the <a href="http://www.measureup.com/">MeasureUp practice exam</a> and hammer it to death.</li>
<li>Go <a href="https://www.microsoft.com/en-us/learning/exam-70-463.aspx">take the test</a>.</li>
</ol>
<div>
<br />
Microsoft even offers their own practice test for this exam if you wish to purchase it. The real exam may or may not include any of the questions compiled in any set of practice questions.<br />
<br />
That said, practice tests are only a means of validating a persons knowledge. The MVA class and Virtual Lab are where the actual learning should take place.</div>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-8835791606205921832016-03-11T13:03:00.001-08:002016-03-12T10:04:53.471-08:00Cloning / Hacking RF remote controls using ArduinoI've been on a smart house kick lately trying to add every sensor I can find to my house that falls within my budget. Due to budgeting issues I am always looking for new ways to do things cheaper.<br />
<br />
Basic sirens coupled with Normal Open and Normal Closed sensors were the obvious first step because they are cheap and fairly easy to understand and implement. And they give you rudimentary detection and feedback capabilities. But I want more, and to get more I have been exploring all the IP based options available for controlling lights, outlets, and other household devices.<br />
<br />
Unfortunately, in part to the current smart house craze, any device that supports traditional smart house technology like z-wave, wifi, and zigbee are extremely expensive. But then I thought about traditional remote controls.<br />
<br />
The vast majority of remotely controlled devices use standard Radio Frequencies (RF), usually in the 315 or 433MHz range. Most people do not consider this to be smart house related because they only work up to a range of a few hundred feet, are usually unidirectional in communication, have no interface to be controlled through the Internet, and absolutely zero security around the protocol.<br />
<br />
There isn't much I can do about the lack of security, that's just a fact of life about cheap remote controls. However, I did not see any reason why I could not make a computer connected radio that sent out the same wave patterns as my remotes. With these cloned signals I would then be able to remotely and centrally control anything with an RF receiver in it.<br />
<br />
After days of reading about the subject and other peoples attempts to do the same thing, I bought an arduino and a 315MHz receiver/transmitter set. Three days later after a lot of signal analysis, trial and error, I finally got my first signal cloned. I now have the ability to repeat the process for any RF remote that operates on the 315MHz frequency with a fixed code and control its item remotely as a fully integral part of my smart house.<br />
<br />
The process involved:<br />
- writing a program for arduino that sampled one of the analog pins on a tight loop tracking the strength and duration of the signal fluxuations.<br />
- I then moved this data into excel where I normalized it to determine the binary code it was sending.<br />
- Converted that binary code into an integer mapping table for condensed storage.<br />
- Wrote a program to use a digital out pin to send that signal to the transmitter which turned it back into an analog wave form in the air.<br />
<br />
To be fair, both of the base programs I started with someone else who knew the arduino far better wrote, I just modified them to be more efficient for my use case.<br />
<br />
Here is the code for the RF Receiver to capture signals:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: 450px; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> #define analogPin A0 // RF data pin = Analog pin 0
byte dataBuffer[512];
int dataCounter = 0;
int maxSignalLength = 255;
const unsigned int upperThreshold = 100;
const unsigned int lowerThreshold = 80;
void setup() {
Serial.begin(115200);
// wait until a LOW signal is received
while(analogRead(analogPin) < 1) { }
// got HIGH; read the rest of the data into dataBuffer
for (int i = 0; i < sizeof(dataBuffer); i = i+2) {
// LOW signals
dataCounter = 0;
while (analogRead(analogPin) > upperThreshold && dataCounter < maxSignalLength)
dataCounter++;
dataBuffer[i] = dataCounter;
// HIGH signal
dataCounter = 0;
while(analogRead(analogPin) < lowerThreshold && dataCounter < maxSignalLength)
dataCounter++;
dataBuffer[i+1] = dataCounter;
}
Serial.println("LOW,HIGH");
delay(20);
for (int i = 0; i < sizeof(dataBuffer); i = i+2) {
Serial.print(dataBuffer[i]);
Serial.print(",");
Serial.println(dataBuffer[i+1]);
delay(20);
}
}
void loop() { }
</code></pre>
<div>
<br /></div>
<div>
Receiving the signals is pretty straight forward. You have to wire your receiver into the arduino. If you are holding it with the top of the board facing you, then the pins are from left to right: Power, Data, Data, Ground. Hook the Power (or Vcc) into a 5V pin, 3.3V should work as well. This code is supposed to be used with the second Data pin, but it doesn't really matter, the HIGH/LOW values will just be reversed if you use the other pin.<br />
<br />
Upload the code to the arduino, it will automatically run since it is all in the setup function. Press Ctrl+Shift+M to bring up the serial monitor, and make sure you set it to receive at 115200.<br />
<br />
Immediately press the button on your remote. Once the data appearing on the screen it is too late, the capture has already taken place. I had to do this process several times, playing with the data each time until I finally got a capture that was clean enough I was able to find a good pattern in it.<br />
<br />
Next you need to paste the data into excel, or some other text editor and start manipulating it. You need to split it into two columns, a HIGH and LOW column, then count the number of occurrences of every value one column at a time. I wrote a macro to do this for me (the guy who's tutorial I followed wrote a php script for himself).<br />
<br />
Next sort the value low to high based on the value that was being counted.<br />
<br />
This is the really tricky part now. You need to identify the signal groupings. Most basic remotes are only going to have two. A common short burst and a common long burst; or smaller number and larger number. So pick the low value and the high value that occur most frequently and group the other more random values in on those two. I picked the values manually, but had Excel re-write the originally values into two new "normalized" columns for me.<br />
<br />
Then you re-combine the two new "normalized" columns back into a single column, perform a Unique function on them, and number each unique value. The purpose for this is we need an easy way to identify the values, copying and pasting all those values pairs into code over and over would just get too difficult very fast.<br />
<br />
Finally you make another column which pairs the two "normalized" columns up with their newly assigned unique value. Combine all those values into a comma delimited string. And start searching for patterns. I often find that the unique value column is a better place for me to go pattern hunting in. Values of 0 and 255 are timeouts and can usually be ignored; if they happen in a very specific pattern inside of the pattern you end up identifying then it could indicate a missing value that you need or could try guessing at.<br />
<br />
Here is what my excel sheet looked like through one of these processes:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://2.bp.blogspot.com/-eS9Yd5PAr1I/VuO6YN27g8I/AAAAAAAAwqw/Z7kXXxXh3IcNOrpd-tZvTX2bI_dNReUEw/s1600/RFSignalHack.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="312" src="https://2.bp.blogspot.com/-eS9Yd5PAr1I/VuO6YN27g8I/AAAAAAAAwqw/Z7kXXxXh3IcNOrpd-tZvTX2bI_dNReUEw/s640/RFSignalHack.png" width="640" /></a></div>
<br />
<br />
<br /></div>
<div>
And here is the Code to re-transmit the signal, however it will have to be modified for your values:</div>
<div>
<br /></div>
<div>
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: 450px; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> #define transmitPin 7 // RF transmit pin = Digital pin 7
int ding[] = {1,2,3,2,3,2,3,2,2,2,2,2,3,2,3,2,3,2,2,3,3,2,2,2,2};
// Play with this value
int timeDelay = 155;
int signalPin = 13;
int numberOfTransmissions = 19;
int numberOfSignalInTransmission = (sizeof(ding)/sizeof(int)) - 1;
void setup() {
Serial.begin(9600);
pinMode(transmitPin, OUTPUT);
pinMode(signalPin, OUTPUT);
delay(2000);
// wait 3 seconds, then transmit the payload
int highLength = 0;
int lowLength = 0;
for(int y = 105; y < 415; y=y+10) {
for(int x = 0; x < numberOfTransmissions; x++) {
if(x % 2 == 0)
digitalWrite(signalPin, HIGH);
else
digitalWrite(signalPin, LOW);
for(int i = 0; i < numberOfSignalInTransmission; i++) {
switch(ding[i]) {
case 1:
lowLength = 3;
highLength = 119;
break;
case 3:
lowLength = 3;
highLength = 11;
break;
case 2:
lowLength = 11;
highLength = 3;
break;
case 4:
lowLength = 5;
highLength = 5;
break;
case 5:
lowLength = 42;
highLength = 2;
break;
}
// transmit HIGH signal
digitalWrite(transmitPin, HIGH);
delayMicroseconds(highLength*y);
// transmit LOW signal
digitalWrite(transmitPin,LOW);
delayMicroseconds(lowLength*y);
}
}
}
}
void loop() { }
</code></pre>
</div>
<div>
<br />
The transmitter comes with 3 or 4 pins depending on if it has an onboard antenna or not. The antenna is absolutely critical; I had a receiver that came without one and it would rarely pick up any signals unless I held a bit of mettle over the solder point where the antenna was supposed to be.<br />
<br />
The pins from left to right if you are holding it facing you are: Digital receiver (hook to digital pin 7 for the code above), Power (Vcc 3.3V or 5V, they can usually go up to 12V for increased range), Ground.<br />
<br />
The code above is designed with three nested for loops.<br />
<br />
The inner most simply takes the ding[] array, converts it back into the values from the normalized columns in your excel document using a case statement, and transmits them. I did not need the last two case statements in this instance.<br />
<br />
The next one out specifies how many times you want to send the pattern; most remotes transmit a repeating pattern several times so there is no point is storing multiple copies of the same pattern if we can just loop through it several times.<br />
<br />
The outermost loop is very important. It adjusts the length of the signals you are sending out, how long each high/low signal should be held for before moving on to the next. The remotes I have seen use a value in the high 200s or low 300s for this. Finding the precise number can be quite difficult though, so looping through a range that increments by 10 microseconds is a much faster way to at least figure out if you are on the right track before trying to narrow the signal down.<br />
<br />
------------------------<br />
Unfortunately it looks like the website I used to figure this all out has gone offline, which is a bummer because it had very detailed step-by-step instructions and was the only set of instructions I could find that I was able to understand clearly enough to figure this all out.<br />
<br />
I believe this is the <a href="http://theforce.dk/rf/">original site</a> I got my tutorial from that is no longer active.<br />
However, here is <a href="http://arduinobasics.blogspot.com.au/2014/06/433-mhz-rf-module-with-arduino-tutorial.html">another site</a> that looks almost as good.</div>
Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0tag:blogger.com,1999:blog-703101161281072462.post-51096447320405949242016-01-13T20:33:00.003-08:002016-01-13T20:33:36.505-08:003D printer - Prusa i3 - Repetier - Basic ConfigurationWhen I first built my Prusa i3 I got lucky enough to find someone who had uploaded a pre-configured copy of the repetier firmware. I uploaded his firmware to my printer and it just worked, like magic.<br />
<br />
Unfortunately I ran into a hickup when helping my son with his arduino based RC car. I accidentally uploaded his arduino program to my printer. And when I looked for the pre-configured firmware again I was unable to find it.<br />
<br />
So, I downloaded the latest and greatest repetier firmware with all default parameters, uploaded it, and kept my fingers crossed. After about a week of experimentation I realized that my end stops were the only thing not functioning correctly, they seemed to be placed on the wrong side of the axis for some reason.<br />
<br />
Another week of playing with settings and no success I finally ran <a href="https://www.reddit.com/r/Reprap/comments/282gfg/prusa_i3_marlin_keeps_going_when_endstop_is/">across this post</a>. All this time I had been focused on my Endstop configuration, it had never occurred to me that was not where the problem lay.<br />
<br />
I jumped down to the XYZ movements area, located :<br />
#define INVERT_X_DIR 0<br />
#define INVERT_Y_DIR 0<br />
#define INVERT_Z_DIR 0<br />
<br />
and changed them to:<br />
#define INVERT_X_DIR 1<br />
#define INVERT_Y_DIR 1<br />
#define INVERT_Z_DIR 1<br />
<br />
And I was back in business, everything was working they way it was before. This time I backed up my configuration in case this happens again.Jeremyhttp://www.blogger.com/profile/11674418943047372077noreply@blogger.com0