Making REST Calls Retryable in Biking Weather Suitability Forecast Application

One frustration I had noticed with my Biking Weather Suitability Forecast Application was that I found it often would not show results until I reloaded the page, and a look at the application logs showed that the REST calls to one of the APIs called had failed or timed out. After doing some research into using Spring’s Retryable options for methods, I decided to make the DailyReportCollectionService method getCurrentDailyReports() retryable, so that it would be attempted a second time after a one second pause if the first try didn’t succeed in returning complete data, with a third and final attempt made as the “Recover” option.

First, I added the following two required dependencies to the dependencies section of the application’s pom.xml file.

        <dependency>
            <groupId>org.springframework.retry</groupId>
            <artifactId>spring-retry</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-aspects</artifactId>
        </dependency>

I needed to add a new Exception, IncompleteForecastException, which would trigger retries of the failed method call. I created a new “exception” package, with the following package-info.java to document the package.

/**
 * Package containing custom Exceptions for biking weather application.
 */
package biz.noip.johnwatne.bikingweather.exception;

The IncompleteForecastException class is a very simple extension of the Exception class, with the following code.

package biz.noip.johnwatne.bikingweather.exception;

/**
 * Exception thrown if the forecast information returned has either less than
 * seven days' data, or missing sunrise or sunset information.
 *
 * @author John Watne
 *
 */
public class IncompleteForecastException extends Exception {
    private static final long serialVersionUID = 1L;

    public IncompleteForecastException(final String message) {
        super(message);
    }
}

I then made the changes required to have the retryable code in DailyReportCollectionService. Since the existing getCurrentDailyReports() needed to be executed both as the standard Retryable code and as the Recovery code, it’s logic was moved to a new private method, appropriately named getRetryableCurrentDailyReports(), starting as follows.

    /**
     * The shared code of {@link #getCurrentDailyReports()} and
     * {@link #getFInalCurrentDailyReports(IncompleteForecastException)}.
     *
     * @return the latest {@link DailyReportCollection}.
     */
    private DailyReportCollection getRetryableCurrentDailyReports() {
.
.
.
.

I added a couple String constants for possible error return values passed on from getRetryableCurrentDailyReports().

    private static final String MISSING_SUNSET_DATA_FOR_TIME =
            "Missing sunset data for time ";
    private static final String MISSING_SUNRISE_DATA_FOR_TIME =
            "Missing sunrise data for time ";
    private static final String MISSING_FORECAST_DATA = "Missing forecast data";
    private static final String NO_DAILY_REPORT_COLLECTION_OBTAINED =
            "No DailyReportCollection obtained";

The existing getCurrentDailyReports() method was revised to be a Retryable caller of the extracted getRetryableCurrentDailyReports(), throwing an IncompleteForecastException with one of the above messages when needed.

    /**
     * Makes calls to get and return the latest {@link DailyReportCollection}.
     * If forecast data is incomplete, throw an
     * {@link IncompleteForecastException} and attempt again in a second.
     *
     * @return the latest {@link DailyReportCollection}.
     * @throws IncompleteForecastException
     *             if the DailyReportCollection to be returned is incomplete.
     */
    @Retryable(value = {IncompleteForecastException.class}, maxAttempts = 2,
            backoff = @Backoff(delay = 1000))
    public DailyReportCollection getCurrentDailyReports()
            throws IncompleteForecastException {
        final DailyReportCollection reports = getRetryableCurrentDailyReports();

        if (reports == null) {
            LOGGER.warn(NO_DAILY_REPORT_COLLECTION_OBTAINED);
            throw new IncompleteForecastException(
                    NO_DAILY_REPORT_COLLECTION_OBTAINED);
        } else {
            final SortedMap<Long,
                    DailyAndHourlySuitabilityForDay> dailyReports =
                            reports.getDailyReports();

            if (dailyReports.size() < 7) {
                LOGGER.warn(MISSING_FORECAST_DATA);
                throw new IncompleteForecastException(MISSING_FORECAST_DATA);
            }

            for (Entry<Long,
                    DailyAndHourlySuitabilityForDay> entry : dailyReports
                            .entrySet()) {
                final DailyAndHourlySuitabilityForDay suitabilityForDay =
                        entry.getValue();
                final Long dayValue = entry.getKey();

                if (!StringUtils.hasText(suitabilityForDay.getSunrise())) {
                    LOGGER.warn(MISSING_SUNRISE_DATA_FOR_TIME + dayValue);
                    throw new IncompleteForecastException(
                            MISSING_SUNRISE_DATA_FOR_TIME + dayValue);
                }

                if (!StringUtils.hasText(suitabilityForDay.getSunset())) {
                    LOGGER.warn(MISSING_SUNSET_DATA_FOR_TIME + dayValue);
                    throw new IncompleteForecastException(
                            MISSING_SUNSET_DATA_FOR_TIME + dayValue);
                }
            }
        }

        return reports;
    }

I then added the new recovery method, getFInalCurrentDailyReports(final IncompleteForecastException e) [sic], making the third and final attempt to get results, if needed.

    /**
     * Final attempt at getting complete data for the
     * {@link DailyReportCollection} that was to be returned by
     * {@link #getCurrentDailyReports()}. This time, do not check for
     * completeness of data, and just return results obtained.
     *
     * @param e
     *            the last {@link IncompleteForecastException} thrown by the
     *            last retry of {@link #getCurrentDailyReports()}.
     * @return the latest {@link DailyReportCollection}.
     */
    @Recover
    public DailyReportCollection
            getFInalCurrentDailyReports(final IncompleteForecastException e) {
        LOGGER.warn(
                "Final attempt at getting current daily reports - not checking for completeness afterward!");
        return getRetryableCurrentDailyReports();
    }

With the additional information provided in the message contained within any IncompleteForecastException thrown, the ForecastController class’ getForecast(model) method was slightly modified to write to the error log if the call to the DailyReportCollectionService’s getCurrentDailyReports() method resulted in such an Exception being thrown.

    @GetMapping("/")
    public String getForecast(final Model model) {
        model.addAttribute("cityname", cityName);

        try {
            model.addAttribute("dailyReports",
                    dailyReportCollectionService.getCurrentDailyReports());
        } catch (IncompleteForecastException e) {
            LOGGER.error("Errored out of all attempts to get forecast", e);
        }

        return "forecast";
    }

Finally, the EnableRetry annotation was added to the BikingweatherApplication class as part of the configuration required by the application to retry any Retryable code.

@EnableRetry
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class})
@ComponentScan(basePackageClasses = {ForecastController.class,
        DailyAndHourlySuitabilityForDay.class,
        DailyReportCollectionService.class, LoggingAspect.class})
public class BikingweatherApplication extends SpringBootServletInitializer {
.
.
.
.
}

After making these additions, I have found that needing to reload the page to see results occurs much less frequently. I hope others may find this retryable code capability valuable.

Adding Verbal Warnings and Time Quotas Configurable by Account for Forced Logouts in Programmatic MacOS Parental Controls

Building upon the original Restoring Forced Logouts Removed from MacOS Parental Controls, the original refinements, and the New Year updates to them, I have since added code to add the following features.

  1. Give two verbal warnings about (approximate) time remaining before forced logout; and
  2. Make the minutes allowed per day a configurable property which can be customized for individual accounts.

The first was added as a courtesy to our boys. The second was added to allow cutting back or eliminating hours for each boy individually if they didn’t complete their list of basic daily tasks to be done to be allowed the full amount of computer time for the following day, or to increase it for sick days and the like. Making it a property also allows changing it on the fly, without having to alter the code, and also eliminates some very inconvenient hard-coded values.

The properties were added to the end of the application.properties file. At the same time, I added a missing JPA hibernate dialect property that was needed to get the application to work properly with my MariaDB instance.

spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQLDialect
# Map properties
users.minutes.[kid-login-1]=60
users.minutes.[kid-login-2]=60

Here, [kid-login-1] and [kid-login-2] are the login ID’s for our boys. The default (for us) values of 60 minutes are set for both accounts.

These properties are mapped to a new ConfigProperties.java class in the model package for the application, which looks like the following.

package biz.noip.johnwatne.logtimer.model;

import java.util.Map;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;

@Configuration
@ConfigurationProperties(prefix = "users")
public class ConfigProperties {
    private Map<String, Long> minutes;

    public Map<String, Long> getMinutes() {
        return minutes;
    }

    public void setMinutes(Map<String, Long> minutes) {
        this.minutes = minutes;
    }
}

We see that the “users” prefix in the “ConfigurationProperties” annotation for the class indicates that those properties in the application.properties file that start with “users.” specify values to use within this configuration class. The configuration class contains a single attribute, “minutes” which maps String values for login IDs to Long values for minutes per day allowed. This ties to the users.minutes.[username]=[minutes] properties in the file. The username part of the property is added as a key to the Map, and the value assigned to the property is set to the corresponding value in the Map.

With the new ConfigProperties class and corresponding properties added, the remaining changes were made to the LogtimerRunner class.

First, the ConfigProperties bean needed to be added as an attribute in LogtimerRunner as follows:

    @Autowired
    private ConfigProperties configProperties;

Then, the logoutIfKidLoggedInTooLong method was modified to

  1. replace the switch statement with hard-coded cases for each child’s name with an iteration of all properties defined in ConfigProperties.getMinutes(), and
  2. Add a new call to the standard MacOS “say” command to notify of time left when there are 10 or fewer minutes left until the forced logout.

The updated method is as follows.

    /**
     * Logs out the specified user if
     * <ol>
     * <li>they are a named child within the family, and</li>
     * <li>the number of minutes they have been logged in exceeds the maximum
     * limit.</li>
     * </ol>
     *
     * @param user
     *            the user whose login time is being checked.
     * @param minutes
     *            the number of minutes the user has been logged in.
     * @throws IOException
     *             if an I/O error occurs.
     * @throws InterruptedException
     *             if an thread is interrupted.
     */
    public void logoutIfKidLoggedInTooLong(final String user,
            final long minutes) throws IOException, InterruptedException {
        if (configProperties != null) {
            final Map<String, Long> map = configProperties.getMinutes();

            if (map != null) {
                Long maxMinutes = map.getOrDefault(user, null);

                if (maxMinutes != null) {
                    LOGGER.debug("maxMinutes for " + user + ": " + maxMinutes
                            + "; minutes: " + minutes);

                    if (minutes > Math.max(maxMinutes - 1, 0)) {
                        LOGGER.info("Logging out user " + user);
                        ProcessBuilder processBuilder = new ProcessBuilder(
                                "/bin/sh", "-c", "./logout-user.sh " + user);
                        processBuilder.directory(new File("/Users/John"));
                        Process process = processBuilder.start();

                        try (final BufferedReader reader =
                                new BufferedReader(new InputStreamReader(
                                        process.getInputStream()))) {
                            String line;

                            while ((line = reader.readLine()) != null) {
                                LOGGER.info(line);
                            }

                            LOGGER.debug(Integer.toString(process.waitFor()));
                        }
                    } else if (minutes > Math.max(maxMinutes - 11, 0)) {
                        final long minutesToGo = maxMinutes - minutes;
                        final String notification = "Logging out user " + user
                                + " in about " + minutesToGo + " minutes";
                        LOGGER.info(notification);
                        ProcessBuilder processBuilder = new ProcessBuilder(
                                "/bin/sh", "-c", "say '" + notification + "'");
                        Process process = processBuilder.start();
                        LOGGER.debug(Integer.toString(process.waitFor()));
                    }
                } else {
                    LOGGER.debug("Time limits not enforced for user " + user);
                }
            }
        } else {
            LOGGER.error("ERROR!! UNABLE TO READ CONFIGURATION PROPERTIES");
        }
    }

I also fixed an off-by-one error in checking the count of login processes for each user in the following method. The value of getCountOfLoginProcessesForUser(persistedLogin.getUsername()) is now checked if < 1, rather than < 2.

    /**
     * Sets the logout time for users no longer logged in, if not already set,
     * to the specified logout time.
     *
     * @throws IOException
     *             if an I/O error occurs.
     */
    private void setLogoutTimesForUsersNotLoggedIn() throws IOException {
        // Do final pass, setting logout times for users no longer logged in.
        for (Logins persistedLogin : loginsRepository.findAll()) {
            if ((persistedLogin.getLogout() == null)
                    && (getCountOfLoginProcessesForUser(
                            persistedLogin.getUsername()) < 1)) {
                // User logged out but not yet indicated as such, so set logout
                // time to now.
                LOGGER.info("*** " + persistedLogin.getUsername()
                        + " logged out; setting logout to now");
                persistedLogin.setLogout(LocalDateTime.now());
                loginsRepository.save(persistedLogin);
                LOGGER.info("*** Record updated: " + persistedLogin);
            }
        }
    }

Finally, to reduce the amount of logging by eliminating info-level logging I had used during initial development, I changed the calculation of lines read from the BufferedReader for the results of the call to get the count of login processes for user in the method getCountOfLoginProcessesForUser(final String user). I removed the logging of each line of output by simplifying

lines = reader.lines().peek(e -> LOGGER.info(e)).count();

to

lines = reader.lines().count();

I hope there might be others who might find these changes to add flexibility to the program helpful. I may consider adding this to GitHub at some point.

Website FINALLY Adapted to Apple Silicon

Back in March, I took advantage of a sale at Costco, advanced the inevitable update, and bought a blue-colored iMac. With its Apple M1 chip, after reinstalling essential packages using Homebrew, I had to make multiple changes to the Apache and Tomcat server configurations to allow my website, and the Tomcat server serving the biking weather suitability forecast application, to work properly again. Since these explorations – including a dead end trying to switch from using Apache to Nginx for the web server, given what seems to be the trend and the latter’s apparent strength being reverse proxy work – were done very sporadically, it took me until this month finally to get it right.

One thing I am glad I did relatively early in the process was to make the new configuration folders git repositories, so I could review the history of document changes and, more importantly, reverse them if needed. This was done using the following steps for the Apache (httpd) folder:

cd /opt/homebrew/etc/httpd/
git init
git checkout -b starting_m1_config
git add --all
git commit -m 'Initial commit of starting M1 config'

The first part of the update was dealing with the change of all /user/local/etc/httpd paths to /opt/homebrew/etc/httpd. After copying my backed-up configuration files from /usr/local/etc/httpd to /opt/homebrew/etc/httpd, I did a global search and replace for this path change throughout the files in the updated folder.

I uncommented a few modules and updated the php_module path reference in httpd.conf:

LoadModule xml2enc_module /opt/homebrew/Cellar/httpd/2.4.53/lib/httpd/modules/mod_xml2enc.so
LoadModule proxy_html_module /opt/homebrew/Cellar/httpd/2.4.53/lib/httpd/modules/mod_proxy_html.so
.
.
.
LoadModule proxy_connect_module /opt/homebrew/Cellar/httpd/2.4.53/lib/httpd/modules/mod_proxy_connect.so
.
.
.
LoadModule php_module /opt/homebrew/Cellar/php/8.1.6/lib/httpd/modules/libphp.so

Finally, I moved the ProxyPass and ProxyPassReverse parameters from httpd.conf to the two <VirtualHost *:443> elements – the only difference between the two being one with a ServerName of localhost, the other johnwatne.no-ip.biz – within the extra/httpd-vhosts.conf configuration file. I also added additional proxy-related parameters based on suggestions in various items found in searching for answers. Whether or not they were needed, I don’t know, but they are part of my final, working solution:

   ProxyPreserveHost Off
   ProxyRequests Off
   ProxyVia Off
   <Proxy *>
      Require all granted
   </Proxy>
   ProxyPass "/bikingweather" "http://localhost:[tomcat-port]/bikingweather"
   ProxyPassReverse "/bikingweather" "http://localhost:[tomcat-port]/bikingweather"

In the code above, [tomcat-port] is the port on which the Tomcat server is listening, which is 8080 by default. The final thing I needed to do with the Apache configuration was to make sure I was consistent in having both paths listed for the ProxyPass and ProxyPassReverse parameters not end with a “/” character. Having only one of them end with the additional character was breaking the proxy URL reference, and I kept getting whitelabel error pages from Tomcat.

One item I needed to fix for the update to Apple Silicon on the web application side was to update one of the dependencies used by the biking suitability forecast application, a Spring Boot application. Looking through the logs for the application when loading the application, I saw the following [edited to remove more info than is likely helpful]:

2022-05-29 23:04:44,238 ERROR nsServerAddressStreamProviders[line  73] Unable to load io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS.
java.lang.reflect.InvocationTargetException: null
	at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:79) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:483) ~[?:?]
	at io.netty.resolver.dns.DnsServerAddressStreamProviders.<clinit>(DnsServerAddressStreamProviders.java:64) ~[netty-resolver-dns-4.1.76.Final.jar:4.1.76.Final]
.
.
.
Caused by: java.lang.UnsatisfiedLinkError: failed to load the required native library
	at io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider.ensureAvailability(MacOSDnsServerAddressStreamProvider.java:110) ~[netty-resolver-dns-classes-macos-4.1.76.Final.jar:4.1.76.Final]
	at io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider.<init>(MacOSDnsServerAddressStreamProvider.java:120) ~[netty-resolver-dns-classes-macos-4.1.76.Final.jar:4.1.76.Final]
	at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67) ~[?:?]
	... 94 more
Caused by: java.lang.UnsatisfiedLinkError: could not load a native library: netty_resolver_dns_native_macos_aarch_64
	at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:224) ~[netty-common-4.1.76.Final.jar:4.1.76.Final]
.
.
.
	... 91 more
	Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_resolver_dns_native_macos
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:224) ~[netty-common-4.1.76.Final.jar:4.1.76.Final]
.
.
.
	Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_resolver_dns_native_macos.jnilib
		at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:166) ~[netty-common-4.1.76.Final.jar:4.1.76.Final]
		... 99 more
		Suppressed: java.lang.UnsatisfiedLinkError: no netty_resolver_dns_native_macos in java.library.path: /Users/John/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
			at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2434) ~[?:?]
.
.
.
Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_resolver_dns_native_macos_aarch_64.jnilib
	at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:166) ~[netty-common-4.1.76.Final.jar:4.1.76.Final]
.
.
.

A little bit of digging revealed that these were related to using the Intel version of the netty-resolver-dns-native-macos Maven artifact. To use the correct version, I needed to add it to the dependencies, along with the dependency-specific “classifier” value to ensure I was using the Apple Silicon – aarch_64 – version of the resolver. This was done by adding the following to the list of dependencies in the application’s pom.xml file.

        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-resolver-dns-native-macos</artifactId>
            <classifier>osx-aarch_64</classifier>
        </dependency>

I also updated the netty.version value in the properties section of pom.xml to the current version.

After all these fixes, the main WordPress site’s and the web application’s links to each other – including shared font files – are once again working correctly.