Hobby Project: Moving to a Range of Articles
Tags: ranges
Published: 2020-02-09 07:44:24

These next changes came with some architecture changes, which were too intrusive.

https://gitlab.com/jessephillips/devarticlator/-/compare/v0.0.7...v0.0.8

```diff
- private Paths articlePaths(string folder, ArticleMeta am)
+ private const(Paths) articlePaths(string folder, const ArticleMeta am)
```

I marked method parameters with `const` and in the case of this method, my unittesting was triggering a warning because the call would produce no side effects. 

```dlang 
auto suppressWarning = articlePaths("someLocation", meta);
```

By assigning to a variable the compiler was satisfied that I was planning to do something with the result. 

Thinking more on this for the article, I should have returned immutable as the type is unique and now `const`, effectively making it immutable anyway.

--------

https://gitlab.com/jessephillips/devarticlator/-/blob/v0.0.8/source/devtopull.d#L78 

I've also implemented the initial interface for making the Dev api into a range. The first part was to remove the dependency of making a web request. 

```dlang 
private const(ArticleMe)[] function(uint page) @safe articlePage;
```

I did this by using a function pointer which provides a list based on a page number. I can now use this for testing my range without the web request. 

Originally I was considering the use of [std.concurrency](https://dlang.org/phobos/std_concurrency.html) which would allow me to delegate to a different thread, but ultimately did not feel like threading was of benefit over the function pointer.

------

Inside my function I create a local variable which will hold the web request results. This is necessary because vibe.d use a delegate for its own processing, it is also why I thought the message passing would be helpful. 

```dlang
const(ArticleMe)[] ret;

    requestHTTP(format!"https://dev.to/api/articles/me/all?page=%s"(page),
 ```

One thing I don't know is how many pages, guess 404 is how I will know to stop. 

------

https://gitlab.com/jessephillips/devarticlator/commit/9506d04668a49d33afe37ef252018670dcb6a1bc

One thing I needed to change was to modify the save article to operate on a single element instead of a range. This wasn't hard because I already had implemented as a nested function. But I did forget to grab the `in` contract.

This all lead to:

```dlang

alias inPreviousPull = (x) => !x.isNewerArticle(ps);

auto newestArt = devtoMyArticles()
        .until!inPreviousPull
        .structureArticles
        .tee!(x => x.saveArticle(savePath))
        .reduceToNewest;
```

Let's compare with my previous post guess at the implication. 

```dlang
devreqArticleRange
  .until!inPreviousPull
  .structureArticles
  .saveArticles(savePath);
```

The primary change here is the driving operation is no longer saving the articles. Instead I would like to get the article which is the last one written so I know when to short circuit the next time. This is important because the api will produce unpublished first so I can't just take the first article. With that said I should modify [`isNewerArticle`] (https://gitlab.com/jessephillips/devarticlator/-/blob/v0.0.8/source/devtopull.d#L34) to ignore unpublished articles.

[`tee`](https://dlang.org/phobos/std_range.html#tee) allows splitting the pipeline execution, this allows me to save each article while I'm finding the newest one.

Generally walking a range multiple times isn't a big deal. Large ranges or in this case where a range would have a larger wait on devices. Though I probably could have just as easily downloaded all the articles into an array, it seems unlikely that even 200 pages would have memory issues in today's computers.

------

I had chosen to start with pulling the articles to become a little familiar with the api and what my article structure might look like. 

Clearly I've gone a little overboard. I could have exported the articles long ago. Instead I'm trying to be more robust and apply applicable attributes.

I can't say that my code was improved because of it though. The annotations were added only as a description of how the function already behaved. That being said, I think my coding style has been heavily influenced by these attributes as I have strived to be able to use them. 

------

I did not write tests first, but obviously have some items to add coverage for.