Synthetics monitoring with New Relic for PowerSchool products

Beginning August 2018, the school district I work for rolled out a new Learning Management System, PowerSchool Learning, for use by faculty, staff and students.  While our primary Student Information System (PowerSchool SIS) is housed on-premises, the learning management module is hosted by the vendor.  One of the concerns brought to my attention prior to the rollout was monitoring i.e. what kind of notice were we to expect during outages, and how could we most effectively communicate that to all effected parties. 

So, to start with we needed an effective way to verify that a client could log in to the self-hosted SIS, then navigate through to the offsite-hosted LMS.  While there are many monitoring products out there, and many are very good at what they do, we eventually settled on using New Relic Synthetic Monitoring.  The primary considerations for that were as follows; the scripted browsers are easily configurable, the built-in tracking for Service Level Agreements, and the ease of use for the alerting system.

New Relic offers a fairly decent overview of scripted browsers here.  The breakdown is that it uses a Node-based scripting language to have a Selenium-powered Chrome browser navigate through a web page.  So, a virtual browser navigates through a desired site, and if it can’t it registers a failure.  Success or failure is then available through a dashboard, and alerts can be set to notify if a site is unreachable.

The implementation of New Relic Synthetics could legitimately be termed as my first real experience with programming.  As such, the process of creating the scripted browsers most likely took me significantly more time than it would have someone with actual programming experience.  Additionally, they are less than tidy and not what anyone would consider efficiently written.  However, the two I have written are functional- that’s the important part, right?  At some point in the future I will be taking another look at them to get them cleaned up.  Currently both scripts log in to our PowerSchool Teacher portal, then navigate to one of the additional modules and check the page for the presence of a completion indicator.  You can take a look at my first programming attempts here. 

After testing the scripts, and setting the run schedule, I moved on to setting up alert channels.  There are multiple types of alert channels you can configure, in my case I chose email.  The next step was to set an alert policy.  New Relic offers a variety of other monitoring, but for Synthetics specifically your only real choice is to alert on failure of a specific check.  Of the available incident preferences, I chose policy since there are only two possible results of a synthetics check: pass or fail.

It didn’t take long for me to realize I needed additional configuration.  While considering when the checks would run, I had not accounted for system maintenance.  As a result, I started receiving a number of what could be considered false positives.    Essentially, when our on-premises systems were having maintenance performed, I would be inundated with incident open/close notices.  The false positives were resolved by configuring monitor downtimes.

While this could be considered to be a non-standard use of a monitoring product, I now have an easily available dashboard with which to monitor the increasingly large number of third-party websites our faculty require access to.  I have plans to make the dashboard generally available to our Help Desk front line operators in the near future.  In addition to a more active monitoring of our third-party site access, the initial troubleshooting burden on the help desk should be significantly reduced.