I thought I had it licked. It made perfect sense even, and I had a rational explanation for my findings - except that they were wrong. What I was noticing was that after I'd reposition the sensor, it would usually work fairly well. I've been mounting the sensor to the inside of the lid now as it has the best, widest, unfettered line of sight (and therefore sound) to the bottom of the tank. I only put one bag of salt in this last time because in order for this sensor to work properly it will have to be calibrated, and in order to calibrate it I need to get a base reading from when the tank is fully empty, and then another when it is brimful of salt. After starting with those two values, a simple percentage can be calculated, and I've got my datapoint. Unfortunately, every time I've walked away from the sensor in the tank, it invariably begins to give more and more sporadic readings until it eventually fails outright. The last couple of tests I did, moving the sensor, getting a [better] position and then hot gluing it in place, it would seem to work well (sometimes VERY well) giving fairly consistent measurement readings. I'd think I had it settled when I'd go to check the readings a few minutes later and the sensor is returning 0 cm. Power cycling didn't seem to help. Scratch head.
I started to wonder if it may be conductivity, or capacitance in the hot glue causing some sort of cumulative effect. I was, after all, getting some of the glue directly on electrical components on the HC-SR04, but I always thought hot glue was non-conductive. This time I tried to mount it by the wires only so it effectively dangled inside the tank, pointing more or less at the bottom. Obviously aiming was a lot more difficult like this, but at this point I was a few steps behind aiming the sensor. Could it also be salt dust in the air? I know salt water is highly conductive... Not sure about salty air. No change. Sensor worked for a while, then eventually pinned back to 2 microsecond round trip times.
Remember, that last go round I took out all the math and just had the sensor send me the round trip time in microseconds. Regardless of where I mounted it, eventually it returned to a round trip time of 2-3 microseconds!
Suddenly it dawned on me, the only way it would be getting such a low reading of distance was if the sound pulse emitting from the transmitter was actually traveling through the PCB and being picked up
physically by the receiver. A theory began to form, I was testing the sensor long before the hot glue had solidified. What if, as the hot glue got cooler, it also became harder and therefore made the whole assembly more rigid, allowing physical transmission of sound better? It would explain why it took a few minutes of decent operation for the sensor to begin to get sporadic readings, and once it was fully hardened - remain consistent at returning 2-3 uS times...
To test this theory, I removed my rigid mount, peeled all the hot glue off the sensor (I've been trying multiple sensors along this time too, by the way. They all seem to behave identically, and none of them ideally.) I found a piece of foam, and just put a dab of hot glue on each corner, careful not to touch any of the circuitry and to only [just barely] hold the sensor. Once mounted, I closed the lit again and the readings were rock solid. It made so much sense in my mind, I had conquered this tricky problem. About half an hour later I checked the readings and the sensor was again returning 2 after 2 after 2. Back to the drawing board.
Could it be orientation? Maybe, for some odd reason, these sensors do not like being mounted facing down? To test this, I detached the lid and just set it on top of the unit, facing upward. Same results. Worked for a while, then got more and more sporadic and eventually dropped and stayed at 2.
I know I had measured power on it before, maybe it's worth checking again. I'm also trying a slight modification to the code wherein I ground the echo pin for a few uS before sending the trigger. Who knows, maybe if there is some capacitive effect at play here that may help reduce the effect?
All this time I am swapping back and forth with two Megas, one in play while the other gets programmed and tested. The more frustrating part is that on the bench, the one [not hooked up to the water softener setup] seems to work reliably, indefinitely, further casting a coin in the coffer of environment as a cause. If environment is to blame though, [what?] Wire length? It's about 8 feet of 4/22. Power? I'll check again. Noise? There shouldn't be any electrical interference inside or near the Water Softener. Operating environment? It seems to do it in open air pointed at the ceiling. This is still very much a mystery. I might have to get someone with a scope in here to see if [together] we can't figure out what on earth is going on with these sensors!