DigitalLabs@MMU

Digital enterprise at Manchester Metropolitan University

You are here: Home / Archives for Our Team

Our Team

By

Seventh Week at Digital Labs

Mistakes Made

This week has been a slow week, through-out development time we constantly focused on reducing any kind of feature creep. The mistake I made this week was focusing too much on the small features leading to specific features taking much longer than expected. Luckily the server, search page and import are 100% complete allowing me to focus on the view page next week.

Ui-router Negates ngRoute

Single Page Web Applications (SPA) contain no page reloading, instead, they download the whole application. To route users to different views (web pages), the Angular ngRoute module can be used. NgRoute functions by linking controllers to views by watching the URL using $location.url(). Using $routeProvider specific routes can be defined, the example below routes ‘/import’ URL to the importController and import.html template.

 

angular.module('importApp',['ngRoute'])
​
  .config(function($routeProvider)){
           $routeProvider
          .when('/import/:importId',{
           templateUrl: 'import.html',
           controller: 'importController',
           resolve: {
           //get data
          }
  })
};

 

You will notice ‘:importId’ appends the import URL the colons tells us that ‘importId is a path parameter, within the importController the URL parameters can be accessed using $routeParams. Additionally, the example uses a resolve property which can be used to request data before instantiating the controller. If the resolve property object contains a promise the controller will only be instantiated once resolving all promises if a rejection happens the $routeChangeError event fires. The $routeChangeError can be hooked upon to redirect a user to a different view.

NgRoute is very powerful tool that can easily support small-scale applications but lacks required features for more complex ones. An alternative to ngRoute is ui-router, ui-router is a 3rd-party module that contains all ngRoute features as well as much needed extra functionality. Ui-router provides multiple named views, one template can contain multiple views, nested views which can inherit from one another. Furthermore, ui-router provides a superior way to access state parameters. The example below shows 2 views within the same template.

 

<!--index.html-->
<body>
   <divui-view='importDetails'></div>
   <divui-view='importSelection'></div>
</body>

 

angular.module('importApp',['ui-router'])
​
  .config(function($stateProvider)){
           $stateProvider
            .state('import',{
             views: {
             'importDetails': {
               templateUrl: 'importDetails.html',
               controller: 'importDetailsController'
            },
             'importSelection': {
              templateUrl: 'importSelection.html',
              controller: 'importSelectionController'
            }
          }
  })
};

 

Each ‘div’ with the ui-view attribute within the html describes each view, the view names must match with the keys within the views object.

Furthermore, ui-router provides lazy loading allowing the application to download resources on-demand rather than startup time. Using lazy loading can drastically reduce the startup time of an application only by downloading the necessary data when needed. Ui-router will download the needed data for a state before entering by waiting for the function promise to resolve. Below the example shows how to configure the import state to lazy load, which loads import.js before going to the ‘import’ state.

 

angular.module('importApp',['ui-router'])
​
  .config(function($stateProvider)){
            varimportState={
            name: 'import',
            url: '/import/:importId',
            templateUrl: 'import.html',
            controller: 'importController',
            lazyLoad: function(){
             returnSystem.import('import')
            }
          }
           
           $stateProvider.state(importState);
  })
};

 

State Page Stored in URL

When first meeting with the client one requirement the client requested was the ability to save searches and share views, each feature would take a large amount of time and may provide limited functionality. Instead of having two different features, save searches and share views, storing the pages state within the URL provides a cleaner solution which would take less development time.

Storing the pages state within the URL, is common practice with Single Page Web Applications (SPA), allowing the user to intuitively share the pages URL knowing the URL with direct back to the exact state of the page.

Storing the page state in the URL is a cyclic process, each variable that needs to be watched within the page is stored within the URL as a query parameter, when one of the watched variables changes the corresponding URL parameter changes, triggering the pages UI to update.

A common mistake when storing the page state in the URL is to update the URL and UI independently. Updating the URL and UI independently is considered bad practice as it breaks the cycle thus the URL and UI could de-sync causing unwanted side effects.

 

To maintain the page state within the URL I used ui-router, using $stateParams and $sate services. The first part I developed was to extract the parameters from the URL using $stateParams. The $stateParams is a service to provide controllers or services with URL parameters which are declared when configuring the application. Within the $sateParams object each key corresponds with each URL parameter with the value the value of the parameter. Within the application, the URL parameters are extracted if a parameter has changed or when the state is initially loaded. The example below demonstrates how to extract the importId URL parameter using the $stateParams service.

 

app.controller('importController',['$stateParams', function($stateParams){
   varurlParams=function(){
       return$stateParams.importId;
  }
}])

 

Once the application could extract URL parameters the second step was to update the parameters dynamically. When researching how to update URL parameters the UI-router official documentation said to use the $state service. The $state service exposes multiple methods to control the current state, for example, $state.get gets the registered StateDeclaration object (used to define a state). To update the URL parameters the .go method can be used passing the absolute state name, state object or relative state path as well as a map of parameters which will populate $stateParams. To update the current state a ‘.’ can be passed as the state name to the .go method.

 

app.controller('importController',['$state', function($state){
   functionstateReload(params){
       $state.go('.',params);
  }
}])

 

The issue with using this method is the application state reloads each time the .go method is called, which doesn’t provide a seamless user experience. The application page reloads because the .go method calls the .transitionTo method which transitions to a new state. To fix the reloading issue the dynamic flag can be used. When declaring parameters for a specific state within the application configuration the dynamic flag can be set to true. Setting the dynamic flag to true will not cause the state to enter/exit when the parameters value changes, when set to false the state will enter/exit when a parameter changes

 

angular.module('importApp',['ui-router'])
​
  .config(function($stateProvider)){
            varimportState={
            name: 'import',
            url: '/import/:importId',
            params: {
              importId: {
              dynamic: true
              }
            },
            templateUrl: 'import.html',
            controller: 'importController',
          }
           
           $stateProvider.state(importState);
  })
};

 

A comment mistake when integrating the page state within the URL is having too many watchers. The $watch method registers a listener callback which is fire whenever the passed expression changes. It might be tempting to attach a listener for each variable when a variable change the .go method is called updating the URL parameters but this considered bad practice. Having too many watchers can impact the application performance drastically, once the number of watchers exceeds around 2,000 an application performance will start to have noticeable performance issues. Instead a function should be implemented and is specifically called when URL parameters need to be updated.

Conclusion

Overall this issues last week have impacted this weeks development before this week started I still had to finish the incomplete features from last week. I was hoping to fully complete the Single Page Web Application (SPA) this week but finishing the Import page took longer than expected. Next week the whole system should be complete leaving me with 1 week of extra development time.

By

Sixth Week at DigitalLabs

Simple Start

The start of the week focused on finishing the import page and testing the search page. This is the third week of development meaning by next week development should be complete, if no more unexpected errors show up.

Errors Everywhere

Development this week has been the slowest, a lot of my time was spent debugging  the code. Unexpected errors will always happen within software development, this week two days were spent fixing bugs. Below details the two major bugs I was experiencing and how I fixed them.

Furthermore, the predicted development time has been pushed back by two days which takes two days off the extra features development time. Luckily development should still easily be completed within the 6 weeks of development time

Third Party Troubles

When building the Single Page Web Application (SPA), I have been using multiple third-party libraries when needed, for example, Angular Materials. Using third-party libraries can save a lot of time allowing me to focus on the application login but it means that my code is tied down to the third-party libraries.

To implement the autocomplete as well as the UI chips, represents a small set of information, I used the Angular jsTag library. The jsTag library relies on angular-typeahead which is a wrapper for twitter-typeahead, I followed the official tutorial to implement all three libraries. Once jsTag was implemented I noticed it didn’t work 100%, when selecting an item using the arrow and return keys the item would not select correctly, only the first character of the selected item would be viewed.

To diagnose the problem I first re-read the official tutorial and documentation double checking my implementation. After confirming I implemented the library correctly, I checked the Issues list on the GitHub repository. The issues contained multiple records of other developers experiencing similar problems but with no solution. Instead of switching to a different a library I attempted to solve the problem myself initially looking at a jsTag demo. When testing the demo application I couldn’t reproduce the same issue but the source code for the application wasn’t available.

Luckily the developer used Github Pages, a service which serves a website from a GitHub repository, to host the demo. I compared the demos source code with mine which again confirmed I implemented the library correctly.

My final step was to replicate the demo exactly from the ground up, testing every time a new part was added. Whilst importing each library I noticed that I was using a different version of the angular-typeahead library. Throughout the documentation and tutorial, there was no mention of version compatibility, the jsTags relies on an earlier version of angular-typeahead. Using the correct angular-typeahead version fixed all the issues I was experiencing.

The demo source code can be found here and the compatible angular-typeahead v0.2.1 can be found here. Additional the issue fix can be found here

Back Button Breaks

The Single Page Web Application (SPA) built with AngularJS, SPAs function similar to desktop applications rather than traditional websites. The advantage SPA have over traditional websites is they have relatively quicker response times, the disadvantage is they can easily break expected features. Whilst building the import page the browser back button provided with unexpected results. The import page contains a simple folder browser when pressing the back button the user would expect the folder browser path to go up, instead pressing the back button went to the previous page.

The unexpected result happens because navigating folders using the folder browser dynamically updates the DOM instead of directing the user to a different URL thus nothing added to the back button’s history.

To fix the back button problem I first researched the problem collating articles on stack overflow and on UI-router. The stack overflow provided many examples explaining how to detect a back button event and how to handle the event but they all relied on the ‘$stateChange’ events which are official depreciated since ui-router 1.0. To migrate ‘$stateChange’ to ui-router 1.0 the Transition Hook API is used instead, after reading the APIs documentation and tutorials I converted the stack overflow example to use the API.

To detect when a transition start runs the onStart hook can be used. This hook can be used to perform tasks before the transition is complete, this hook is also invoked when the user presses the back button. The example below logs to the console ‘Transition onStart Invoked’ when the user switches pages.

$transitions.onStart({},function(transition){
    console.log('Transition onStart Invoked');
})

The problem with using the onStart hook is it doesn’t tell us what invoked the transition change for example by clicking on a link or pressing the back button.I first had to understand what else happens when a transition changes to detect if the used pressed the back button. I discovered that the ‘locationChangeSucess’ event fires followed by the ‘before’ event, which invokes the onStart hook, when the user presses the back button, otherwise the before event fires initially, followed by the ‘locationChangeSucess’. 

Knowing the locationChangeSucces event fires first when the the users pressed the back button, I could hook on to this event. The example below sets a new scope property ‘current location’ to the user’s location when the hook is invoked.

 

 

$scope.$on('$locationChangeSucess',function(){
    $scope.currentLocation  = $location.url();
})

I checked if the currentLocation property was equal to the URL transitioning to, by using the ‘.to‘ method. If both properties are equal, the user has pressed the back button, if not the user has transitioned using a different method. Attaching the hooks to the root scope rather than a local scope will detect the events globally.

The example below logs to the console ‘back button pressed’ and cancels the transition, using the ‘.abort‘ method, when pressing the back button

$rootScope.$on('$locationChangeSuccess', function() {
    $rootScope.currentLocation = $location.url();
});
$transitions.onStart({}, function(transition) {
    //extracts the URL and params from transitioning 
    var newLocation = parseUrl(transition.to());
    
    if(newLocation === $rootScope.currentLocation){
        console.log('back button pressed')
        //stops the transition from happening
        transition.abort();
    }
​
});

The example above uses Angular services to detect a back button click, the same results can be achieved just using pure Javascript. The history object which ”exposes useful methods and properties that let you … manipulate the contents of the history stack”. Using the same hook as before ‘locationChangeStart’ we can use the history object to manipulate the transition. The example below illustrates  how to redirect the user back to the original page 

$scope.$on('$locationChangeStart', function () {
    //as if the user clicked the Forward button
    history.forward(); 
});

The disadvantage with using the history object is that support is unknown for Internet Explorer, Opera, and Safari. Doing a quick test (03/08/2018), Chrome, Safari, Opera and Firefox all support the history object.

Conclusion

This week mostly consisted of debugging but was still no less enjoyable. Seeing the SPA moving slowly closer to completion is immensely gratifying as well as getting feedback from the clients. Next week the SPA should be close to completion if everything sticks to the plan.

By

Fifth Week at DigitalLabs

Another Friday another diary, take a read below.

 

Planning

This week started off the same as last week, developing the backend. When reviewing all the features of the system I realised a few features had been forgotten. Once I completed the forgotten features, I started development on the single page web app (SPA).

Additionally at the start of the week, using Trello I planned out all the tasks I predicted I would complete this week. Planning within software development can be incredibly difficult, errors are hard to predict where and when they will happen. The planning of tasks is used as an indicator rather than a fixed schedule. It’s important to maintain flexibility, throughout the week more tasks could be added and/or re-ordered.

 

Unknown Errors

When building software unexpected errors will inevitably pop up. When performing tests to download data from the OneDrive API an ECONNRESET error triggered. An ECONNRESET error occurs when a TCP connection is unexpectedly closed on the other side, most commonly caused when the other side is overloaded thus kills the connection. Instead of trying to debug the socket, which would result in a lot of wasted time, I took a more logical approach. To fix the error, I eliminated each variable, one by one, until one was left. Continuing to perform more testing I confirmed that the error was occurring due to too many HTTP requests to the same domain.

Initially, I concluded that the OneDrive API was the culprit since I was firing multiple requests in quick succession. To investigate if OneDrive was the cause I read their error responses documentation which lists all the expected errors and their corresponding status codes, discovering the 429 error code. OneDrive API sends a 429 response if overloaded by requests instead of killing the connection thus OneDrive could not be the cause of the ECONNRESET error. Monitoring the network log, when making an HTTP request, the request never reached the API explaining why the API never returned a status code.

Concluding it wasn’t the OneDrive causing the error, I theorized it could be the access point. Each HTTP request passes via a public access point, firing multiple requests in quick succession could trigger the access point to refuse a request in an attempt to throttle a users usage. I switched from a wireless to a wired connection and tested again, the error never occurred.

Even though the ECONNRESET error is incredibly rare I still chose to handle it, by sending a 429 status code back to the client, allowing the client to decide how to handle the request. Figuring the cause of the error did take a substantial amount of time which pushed back the completion date back by a day, luckily we planned for such an event.

 

Async Task

Akin to last week, this week continued to work with NodeJS asynchronous nature. When downloading components from the OneDrive API it can either done synchronously (one by one) or asynchronously (in parallel). Performing the download synchronously ensures the API is never overloaded but is considerably slower, asynchronously is much quicker but can easily overload the server. To achieve each method, both utilise Node JS ‘async/await’ and promises.

To perform each task in order without waiting for each task to complete a simple forEach loop can be used with the anonymous function defined to be asynchronous. In the example below the console.log will fire even if all tasks are not complete.

 

async function asyncTasks(array) {

    array.forEach(async (element) => {
        await task(element);
    });

    console.log('done');
}

 

To wait for all the task to complete a ‘for of‘ statement can be used. A ‘for of’ statement creates a loop iterating over iterable objects such as a string or an array. In the example below the ‘array’ is the iterable object, only when each task is complete with the ‘console.log’ will fire.

async function asyncTasks(array){

    for(const element of array){
        await task(element);
    }
    
    console.log('done');
}

 

To run all the tasks in parallel and wait for the completion of each task a combination of ‘array.map‘ and ‘Promise.all‘ can be used. In the example below the ‘array.map’ creates a new array of the results returned by the function passed in as a parameter. To pass additional parameters to the task function, the last parameter of array.map can be JSON object which can be accessed by the task function with the ‘this’ keyword 

async function asyncTasks(array){
    const promisesToResolve = array.map(task);
    await Promise.all(promises);
    console.log('done');
}

 

Running too many tasks in parallel can be too intensive for the CPU or memory, to stop this from happening you can ‘array.map’ a subset of the whole array.

Utilizing these three methods when building the backend has been incredibly useful, for example using the ‘Promises.all’ method to add multiple components to the MongoDB database. To read more about Javascript ‘async/await’ loops check out the Lavrton Blog.

 

Database Specification

From the ground up the system has been built to allow each component to be swapped out, for example, the file storage could be OneDrive or Google Drive, the frontend could be an SPA or a mobile application. Both the file storage and front-end communicate via an API therefore each one could easily be swapped out, on the other hand, the server hosts the database which makes it tied down with the system.

Taking inspiration from OpenGL, a programming interface to render 2D and 3D vector graphics, I developed a database specification which any developer can use. A specification outlines all implemented functions what parameters they should take in and their result but leaves the implementation of each function to the developer.

OpenGL is entirely a specification, it relies on developers to implement it, this allows it to be cross-language and cross platform. The database specification allows developers to implement any type of database needed with the minimal amount of code changes, making the database as swappable as the front end and file storage.

 

Conclusion

Overall this week has presented itself with a few challenges that took more time to navigate than was predicted. I still continue to improve my programming skills as well as my understanding of asynchronous programming. The next few weeks have been planned out still allowing 2 weeks of extra development time.

By

Fourth Week at DigitalLabs

My fourth week at DigitalLabs and first week of development, take a read.

Programming Starts

One of my passions in life is to code, learning languages from Javascript to C++. Programming has enabled me to take my ideas and bring them to life, from video games to recursive solutions using dynamic programming, sharing them across the world. As Gabe Newell, the founder of Valve said: “The Programmers of today..are the wizards of tomorrow”.

The first 3 weeks of the project consisted of design and features, providing a complete understanding of the project. On the other hand this week I built the initial building blocks of the system utilizing my knowledge of NodeJS and asynchronous programming flow. The reliance on my technical knowledge has increased each week, NodeJS, AngularJS, and MongoDB all had to be fully understood before this week even started. Fortunately, I have used both NodeJS and AngularJS in the past and used my SQL knowledge and documentation to gain a grasp on MongoDB. Surprisingly I thoroughly enjoyed working on the design and features fully fleshing out the system but, it has been splendorous to start to program.

Building the Backend

The objective for this week was to implement the system’s backend which includes authentication, the Import and Browse APIs, the database and connections to the file storage. The first part I tackled was authentication, which was relatively simple to implement. In previous weeks I thoroughly investigated and tested how authentication would work, to implement I simply copied and pasted the code I had written and stitched together with the single page web app (SPA). The file storage authentication was a similar story, stitching previously written code with the SPA. To construct the Import and Browse APIs, akin to authentication, formerly written code was used. Using the work from previous weeks was especially satisfying, reinforcing the significance of design and features, without it both authentication and the APIs would have stolen a considerably larger amount of time.

Constructing the connections to the file storage and the MongoDB database had I read a vast amount of documentation. For this specific build of the system the file storage used will be OneDrive thus the native API is needed. This was my first time working with the OneDrive API, initially, I read the API documentation learning the limitations and features.

Additionally, I read tutorials from various sources, once I felt I understood how the OneDrive API functioned, I implemented it connecting it the import API. When predicting the time each task would need, I thought the database construction would take much longer. Constructing the database was relatively straight-forward since the documentation was clear and concise.

Asynchronous Flow

More traditional languages, for example, C++ and Java are synchronous meaning that each line is executed in-order. NodeJS is asynchronous, permits other lines of code to execute before others have finished. Asynchronous flow is a difficult paradigm to work with, it requires a full understanding of ‘promises’ and ‘async tasks’.

In previous projects to deal with NodeJS asynchronous nature, I used NodeJS promises. Promises allow you to turn asynchronous code to synchronous by waiting for the eventual completion of an asynchronous operation. Once an asynchronous task has been completed the returning value can be checked to lead to other tasks to fire. One situation where promises are used is downloading data from an API. The time it takes to download data from an API is highly variable thus promises where used to wait for the download to complete then perform tasks on the downloaded data.

Working with DigitalLabs I have learned a second superior way to deal with asynchronous tasks, ‘async/await’. Async/await is very similar to promises, they wait for eventual completion of an asynchronous operation but in a syntactically simpler way. Using promises can be very convoluted and hard to track async/await solves this problem by providing a simpler syntax to solve the same problem

Conclusion

This week I learned a great deal especially about asynchronous programming in NodeJS continuing to improve my programming skills. It feels good that the backend system is now fully implemented allowing me to concentrate on the SPA for the rest of the development time.