I like to think I’m a good troubleshooter, but a recent mishap reminded me how important it is to break complex problems down into their component parts. No one will have this specific problem, so take this story as a general lesson.
Long story short, I interpreted the communication failure with the solar panel inverter as a grid issue, but it turned out to be a power issue related to the uninterruptible power supply (UPS) function. To my shame, I didn’t realize this until a solar technician came over to inspect and walk me through the various parts of the system.
In 2015, we installed 18.9 kilowatt-hours of solar panels that provide almost all of the electricity for geothermal heating and cooling of our home and one of our two cars (a 2015 Nissan Leaf). (“Almost All” is due to the fact that we need to predict future usage when sizing the system, and our estimate is a little lower. NYSEG typically charges us a few hundred dollars in late winter after we use up our summer generation credits) .
Overall, the solar panels work well, powering three SMA Sunny Boy inverters (one for each panel array), distributing power throughout the house, and sending excess power to the grid for use by our neighbors. The inverter publishes power generation statistics on a web portal called Sunny Portal, which emails me a summary of the data daily. I don’t care about the exact numbers, but these emails will alert me if something goes wrong. We had inverter glitches, the connection to the portal would drop from time to time, and one day after troubleshooting I forgot to turn on the main switch.
We did not install any battery storage systems that could provide power to the home during a power outage, in part because the system was designed to make outages infrequent and short-lived. We decided to install a standard outlet on each inverter so that any energy generated during a power outage could be used for specific purposes. In the winter this is enough to charge our phones (100-300 watts of power if the panels are not covered with snow, and 5-60 watts from the charger in the summer is enough to charge our garage upright refrigerator) Power supply (power); is 3000-5000 W, the refrigerator consumes only 150 W after a starting peak of about 300 W).
From July 10 to July 16, we had three power outages. The first was during a tornado warning, but it was actually caused by the local highway department doing repairs to our road, raising a section of the road so high that a passing truck got stuck and snapped the top section leading to our house’s electrical wire. Ouch! The other two power outages were caused by severe storms, which are becoming common due to climate change. I grew up on a farm 12 miles from here, watching the summer weather closely with my father, and today’s weather conditions are more unstable, as you can see from the warnings coming in as I write today. Shortly after I wrote this article, we lost power again. It’s getting boring.
After one failure this spring (a new record of 6 failures in 2024), the SMA inverters lost connectivity and I had to run a reset sequence on them to get them to communicate with the portal again. Luckily, Halco, the company that installed them, was happy to tell me what to try via email so I didn’t have to wait for a technician to move the three switches and circuit breakers in the correct order.
After the first recent power outage on July 10th, the inverter lost connectivity again. I think it’s a bad idea! – It was the same problem, so I performed the reset sequence. But on July 11th our power went out again, so I don’t know if my reset worked. I remember about a day later I reset again but the inverter didn’t show up in the portal and I was busy doing things over the weekend and didn’t check again until July 15th. I never got to check to see if the reset worked because we had a power outage that day and it was our longest outage ever at 29 hours.
During the last power outage, I used an inverter outlet for the first time to power our freezer, so I spent most of the day working with the inverter. When unplugged, they only report flow, not output. It took me a while to figure this out and prove to myself that they produced enough that I didn’t have to worry about the refrigerator going dark. I was thinking about connecting the refrigerator to an unused iMac UPS to eliminate the possibility that clouds or darkness would cause power output to drop to dangerous levels, but the refrigerator is not running on the UPS for reasons I still don’t understand. It is possible that the UPS can detect that the refrigerator is not drawing enough power even though it should be able to handle 150W of power.
Once power was fully restored, I performed the reset procedure again on July 17th to no avail. I tried a few more times before contacting Halco. They agreed that I had tried all the basics and promised to send a specialist. Luckily, he had time a few days before the scheduled meeting and quickly told me something I didn’t know: there was some way to knock on the front panel of the inverters to make them cycle through status messages, one of which was shows what their IP address is and thinks they are communicating correctly. Yes, part of the troubleshooting process involves the case, as sometimes happened with early computers.
“There must be something there,” he said, and we went inside to inspect my network equipment. I showed him where the inverter’s network cable connects to the Ethernet switch…why aren’t the activity lights on on the switch? “Maybe it burned out due to a power outage,” he suggested, but I pointed out that it was connected to a UPS. We traced its power cord to a UPS that was connected to the hub of my temperature monitoring system (see “Wireless sensor tags prevent refrigerator crashes,” July 25, 2018), and its activity light was just as dim. The UPS works because the Eero base station that manages my network is online, as is the device that allows me to control the Honeywell thermostat via the iPhone app.
We pulled the UPS out of the dark corner and examined it. That’s when I realized what was happening. This particular APC UPS has one “main” outlet (top left outlet in the picture below) and several “managed” outlets (bottom left). When a device connected to the main outlet enters sleep or standby mode or turns off, the UPS turns off power to the controlled outlet to save power. However, this mode is not enabled by default on this UPS – it can be enabled by pressing the physical “Master Enable” button. But the Eero Pro, the most important network device, plugs into the main outlet. Why not just leave the controlled outlet on? What changed during the power outage that disabled supervised outlets?
I turn off the APC UPS every time there is a power outage to silence the annoying beep. Since it was in a dark corner, I couldn’t turn on the light, so I had to blindly fumble for the power button. (This button lights up, but the power switch usually turns on a separate status light.) During one of the first two power outages, I must have accidentally pressed the main power button before I found the actual power button. The Eero Pro’s power consumption is only about 5W, which is roughly below the threshold required by an APC UPS to power a controlled outlet. This makes sense; If I plug my Mac into a main outlet and turn the power down to 5W, I only need to power the necessary peripherals while I’m using the Mac.
As soon as I took the APC UPS out of master/slave mode, the Ethernet switch and temperature monitoring hub were immediately up and running, and the SMA inverter immediately connected to its portal. Problem solved!
Why didn’t I realize this for myself earlier? In my defense, my Wi-Fi network works, and since my iMac uses both Ethernet and Wi-Fi, I didn’t notice that the disabled Ethernet switch was interfering with the Ethernet in my office. In retrospect, Tonya and I noticed some minor networking issues that may have been related to my office’s extended Eero being forced to use wireless instead of Ethernet for backhaul.
Additionally, two of the series’ four network devices are operational, so there are multiple LED indicators on the rear panel. I even disconnected and reconnected the Ethernet cable from the inverter, so I looked at the Ethernet switch and didn’t notice that it wasn’t working. I often go months without receiving notifications from my temperature control system, so it’s no surprise that I haven’t heard any signals from these sensors in all this time. And I rarely have to control the Honeywell thermostat from my iPhone, so I don’t notice that it’s unplugged (since it’s connected to a disconnected Ethernet switch, even though it has power).
In other words, my troubleshooting was sloppy. Since I previously solved the problem of connecting the inverter one way, I did not go beyond that solution and did not take the upward connection path. Once Halco Technology allowed me to do this, the problem quickly became apparent.
That, dear reader, is the moral of the story. If your first attempt at solving a technical problem fails, break the problem into parts and make sure each part works independently. Every problem will be different, but by starting at one end and eliminating all possible variables, you can figure out the cause.
Every week, you’ll receive tech tips, in-depth reviews, and in-depth news analysis for discerning Apple users. For 33 years, we’ve been publishing member-supported technology news that makes you smarter.
Thanks @ace for a great article. I went through a similar situation and was stuck in the same situation. Most of the time I find that somewhere in the debugging process I made some assumptions about one of the steps but didn’t actually test them. In your case, you thought the UPS was working, but didn’t actually check it. This has happened to me countless times. Thanks for the reminder to check every link in the chain!
By the way, I don’t mean to be creepy, it’s quite strange and inspiring to see your solar and grid setup. After many years of trying to completely hide the various boxes and cables from view, I finally decided to come to terms with it. I think this will help with troubleshooting, especially since the Asus router and fiber optic modem periodically stop communicating with each other.
Our solar installations include batteries and are both simple and complex. Each panel has a microinverter (37 in total) and three more in the battery compartment (left side of the photo).
Troubleshooting in this case involves checking that each rooftop microinverter is communicating with the combiner. I see a long list of all the microinverters on the system portal, but there was one case where the portal refused to communicate with anyone, including the manufacturer. They had to spend some time booting up my phone and then installing a firmware fix that they “hoped” would prevent problems in the future.
I learned how to establish a direct connection to the combiner through its local Wi-Fi portal (we don’t use it for day-to-day work as it also has a cellular modem and an Ethernet connection that goes back to the combiner through the cable rack). I also confirmed that my overall approach to supporting staff with courtesy and patience resulted in a quick resolution to the issue.
Glad you found this question; I would never have guessed this particular thing if a second pair of eyes hadn’t looked at it!
That, dear reader, is the moral of the story. If your first attempt at solving a technical problem fails, break the problem into parts and make sure each part works independently. Every problem will be different, but by starting at one end and eliminating all possible variables, you can figure out the cause.
Let’s expand a bit: I’m that guy on the team when people have exhausted their troubleshooting knowledge and can’t figure out why something isn’t working.
I started asking them basic questions. Has this program been recompiled? Have you reset this database? Is this overlay directed at you? Does your program actually use this test file or is it another file?
When they tell me: “Of course!”, prove it. This is usually when we discover the problem.
Because when you’ve gone through all the usual steps and still can’t find a reason why it’s not working, that means one of those “facts” is wrong. You will have to question all your assumptions.
Your comment about extreme temperatures reminded me of my experience installing a heat pump in the fall of 2022. Our previous heating system was forced hot air, so the installer fed heat pump power into the old ventilation system, saving us a fortune. We noticed that during the winter the heat pump began to push cold air through the system, but the installer told us that this was normal and should not be a problem in a normal winter. Then the Boston area experienced severe sub-zero temperatures (temperatures could reach 15 degrees Fahrenheit in the early morning hours) and heat pumps simply could not pump heat in those temperatures. I never realized how cold it can be when the house doesn’t reach 50°F! I spoke on the phone with the installer and they explained that in severely freezing temperatures, special wiring must be installed to use resistance heating (like a standard electric oven) to adequately heat the heat pump so it can pump heat instead. Cooling capacity. They didn’t do this because they didn’t think it was necessary, but they didn’t expect it to be so cold. We asked them to install special circuits, and, fortunately, we never experienced such a strong cold spell again. But it’s a reminder that extreme weather conditions can push home devices beyond their design range.
We installed a heat pump in the fall of 2022…in the winter the heat pump started pumping cold air through the system…then the Boston area started getting severely below freezing (about 15 degrees F in the early morning hours). and the heat pump simply could not pump heat at such temperatures. I never realized how cold it can be when the house doesn’t reach 50°F!
There is no reason why they shouldn’t install a gas or oil furnace to power your hot air system’s ductwork. Yes, you’ll need a separate compressor for summer use, but in a cold place like Boston, giving up your furnace to buy less equipment isn’t something you should consider. (I know energy companies around the world are trying to promote heat pumps, but it’s a bad idea, just to shift the heating costs from the gas company to themselves, but it’s still a bad idea).
While some modern heat pumps can handle extremely cold temperatures, most cannot and they suffer greatly when it gets that cold.
In fact, it’s crazy that they thought the Boston area didn’t need a resistive coil system. We have them installed in our heat pump systems in Virginia and our weather is never as cold as yours.
The ideal solution (and should be recommended) is a dual-fuel system. In this system you have a traditional gas/oil furnace combined with a heat pump and possibly a resistance coil. So he uses it when the temperature remains high enough for the heat pump to operate efficiently. When it gets too cold it switches to gas/oil. In the event of a malfunction, the coil will act as an “emergency” heat source. This is nothing new – I saw similar advertisements from HVAC companies 20 years ago.
The dual-fuel option costs more than the heat pump (or furnace) itself, but not much more than a furnace plus an air conditioner. You would think that HVAC installers would promote this as it truly is the best of both worlds, but for some reason they usually don’t.
There is a trend toward thermal engines in our Boston suburb, and a local green volunteer helped discuss these options. In 2022, we are eligible for a $10,000 rebate from our local power company, which is just over half of our total bill. They mentioned the possibility of keeping the old heating system as a backup. However we could only get the full discount if we removed the old heating system which was a 35 year old oil burner that was nearing the end of its life and the installer said there was no room for a heat pump and a heat pump. Old heater. (I don’t remember if there was any mention of the cost or possibility of installing new gas backup heating, but I was more interested in getting the full discount.)
The dual fuel option was not part of the city’s environmental programs, but the installer we chose mentioned it.
I agree that they should push the resistive coil heating element. But the temperature in Boston had not dropped to -15 degrees Fahrenheit in a long time, and they decided that we were interested in saving money. Overall, I think this is an unfortunate problem that occurs in the early days of new heating technologies. At some point I may want to consider adding more thermostats to better cool the attic I’m working in, but we can get away with turning off some of the grilles in the system.
Re: Why doesn’t your refrigerator work on a Mac UPS? When the refrigerator compressor starts, the current consumption becomes sudden and excessive. I’m not surprised UPS can’t handle this.
This is truly “maintaining impact”! This reminds me of my first Air Force service around 1965. The Special Projects Division of the Global Meteorological Center is supported by an IBM 7094 mainframe with up to 32 KB of memory and eight half-inch tape drives. It was powered by an IBM 1410, a machine that converted punched cards into input tape and output tape into print, both at fairly high speeds. Visitors are always impressed by the huge, immaculate computer room, but when walking around the equipment, they may be confused by a dirty-looking piece of 4×4 wood sitting in a dark corner. He was found in a nearby car park and nicknamed the “Nuclear Killer”. You see, the 7094 has an oil-cooled magnetic core, and somewhere in the oil there is a small piece of metal that the filter never caught. When it gets stuck in the kernel, it causes parity errors and causes the machine to shut down. The solution is to take a 4×4 and slam the memory box until the piece of metal comes out of its original position and is back in order. Luckily this doesn’t happen very often and I’ve never seen this used in action. But it adds an interesting local legend to the dull military setting.
Adam, thank you. I soak up articles like this like a sponge because troubleshooting is part of my career (like @mschmitt). I have to force myself to think outside the box, which can be challenging since we all do things differently. I learned more tips from your posts.
In 2015, we installed 18.9 kilowatt-hours of solar panels that provide almost all of the electricity for geothermal heating and cooling of our home and one of our two cars (a 2015 Nissan Leaf).
When the refrigerator compressor starts, the current consumption becomes sudden and strong. I’m not surprised UPS can’t handle this.
I test the whole thing using WattsUp Pro and it shows the power consumption and as far as I can tell the refrigerator compressor only starts at about 300 watts, which seems to be within the capabilities of the UPS. But maybe it’s too fast or something?
Alas, we installed it nine years ago, and although I was involved in the planning with the Halco installer, I didn’t know enough about it. I suspect technology has changed and improved since then. But overall we’re happy with how it works.
My CyberPower UPS has a main power off button that you press to turn it on or off, right where you have to pull the plug with your left hand because the female connector is pulled too tight with your right hand. The same goes for pressing. After accidentally turning off all my stuff over and over again, I trained myself not to press that button! In industrial design, this is called bad capability. Essentially, they’ve put a critical but very simple power button into a form factor that’s easy to use casually. This actually put me off buying a second one with the same buttons, even though it was an otherwise great UPS. They still use the same buttons. So stupid!
As far as I know, the starting power of the refrigerator compressor is only about 300 W, which seems to be within the capabilities of the UPS.
I recently purchased a small refrigerator/freezer for my DIY RV and am testing the battery to see how it performs. It has very low power consumption (45W), but even when I used up to a 300W battery, it would drain when starting the refrigerator.
Post time: Sep-02-2024