Any interest in a community-supported pen information app + website?

[jrista]

I know that IAP has their bushings and tubes app, however support for Android has apparently been abandoned. Further, it appears as though the app requires a database download, so updates are slow and can require a hefty data requirement.

I'm curious if people would be interested in an app + web site, that would work on any device, that uses an online database for data primarily, but with the option to save favorite items to the device for use in an offline mode when necessary?

I could do the work to build out the app (would use something called Ionic, so a single app would work on both iOS and Android as well as support being published as a website online), provide an initial database load. However, I figure allowing the community to correct invalid datapoints, or add new kits, or even add support for say kitless pen design ideas, etc. might make an intriguing capability. Might also make it possible for some kind of basic commentary, limited and maybe moderated, so that any useful information community members have about any given kit could also be added.

I figured, kind of like how Wikipedia works, any edits could be community vetted and rolled back with a collective vote, or something similar to help keep the data valid. Further, all versions of the data for any given "kit" would be maintained forever.

Anyway. I'll probably put something together regardless, as I need something...I'm constantly referring to paper printouts because the IAP bushings and tube app isn't up to date for Android. So I'll probably release something at some point regardless. Mostly curious if anyone would be interested in the community aspect, and whether I should invest any effort there.

 

[Curly]

Perhaps you should be talking to Wayne and Jeff about it as I assume you would want to build your app on Wayne's hard work / data and other contributors. You idea could very well be a logical progression of what has been done up until now.

Pete

Edited to add. Edgar being one of the other important contributors.

 

[Wayne]

Jon,

This was always on our plate of wants. We couldn't find anyone that was willing to volunteer.

This would be great. I would hope that you would work with IAP with this effort.

I personally have been collecting pen data since 2009.

There is an update ready to release in chart form now.

 

[jrista]

Wayne,

If you guys are willing to help, that would be great. Starting with an existing set of data would certainly shortcut the process. Any chance you have it in some kind of tabular format, such as CSV, Excel, anything like that? If so, then it would make it easier for me to turn it into a functioning database that I could actually drive a web app/mobile app with.

If you guys can supply the data, then, putting together an app shouldn't really be that hard. This is what I do, day in and day out, so once I get some free time here I'll start plugging away at it.

 

[sorcerertd]

I love the idea. It would be great to have a reference where I could look up a kit and see not only the tubing, drill, and bushing sizes, but also the instruction sheets. The option to view in the browser would help save space for those that don't have a lot of storage on their phones or tablets. I don't know how these things work, but if data usage is a concern for some, having an option to only download info on wifi would be helpful. Most of us would likely be using the app almost exclusively in our workshops where there there would typically be a wifi connection readily available.

You might be asking for trouble if you let anyone add/edit information like a full blown wiki. Obviously, keeping up with edits could also require a lot of work for the developer(s). Hmm. Collection and verification of the data would certainly be a community effort. Fortunately, most of it has been collected already. If setting up with the initial data, if it's a matter of putting that into a spreadsheet, then I can help with that part.

 

[Darios]

the instruction sheets. The option to view in the browser would help save space for those that don't have a lot of storage on their phones or tablets.

Ah-men to a unified location for instruction sheets.
One potential to save storage would be to just link to the existing vendor PDF of the instructions. You run the risk of having to retool if the vendor changes their naming format, but you are at least not on the hook for file storage/retrieval issues on both the front and back ends.

You might be asking for trouble if you let anyone add/edit information like a full blown wiki. Obviously, keeping up with edits could also require a lot of work for the developer(s). Hmm. Collection and verification of the data would certainly be a community effort.

Possibly a two tier system - comments/reviews open to everyone with the ability to edit the core description to a dedicated group?

But yeah, I'd be interested in this as well. Count me in as a volunteer for just about anything short of the coding itself.

 

[bzahn]

This would be a great addition. Wayne's charts have been so helpful, but as an Android user I've never been able to use the current app.

 

[jeff]

What type of DB do you need on the back end? I'd like to see that hosted on our server just so we have a good investment in the project. One of my retirement projects was going to be a similar effort, but I'm not an app developer so I was going to whip up a simple front end for browser access and rely on Wayne and Edgar for continued data updates. I'm somewhat skeptical of crowdsourced data updates, but maybe it works with sufficient oversight and review.

I'm not sure how I feel about discussions intermixed with the kit data. I envision a lot of tangential discussions about design, techniques, tooling, sources, etc.. That's why we're here I'd hate to see good discussions separate from our forum.

Happy to discuss further.

 

[jrista]

What type of DB do you need on the back end? I'd like to see that hosted on our server just so we have a good investment in the project. One of my retirement projects was going to be a similar effort, but I'm not an app developer so I was going to whip up a simple front end for browser access and rely on Wayne and Edgar for continued data updates. I'm somewhat skeptical of crowdsourced data updates, but maybe it works with sufficient oversight and review.

I'm not sure how I feel about discussions intermixed with the kit data. I envision a lot of tangential discussions about design, techniques, tooling, sources, etc.. That's why we're here I'd hate to see good discussions separate from our forum.

Happy to discuss further.

I'll have to think about hosting. There will need to be both a DB server, as well as a server to host the web app itself, and probably an API server as well. Having it all hosted in one place helps...my original thought was AWS. I am actually not sure about hosting costs here....if this ends up popular, it might in fact end up costing more than a little...

What are you guys capable of as far as hosting goes? Could you host a Nest.js server, and a PostgreSQL database? If you could host both of those then I think that would cover all my needs... I could just put the source code in GitHub, open source. The source code, BTW, would be all typescript/javascript, so it would be quite accessible. Will probably use Nx for a monorepo, so that all the code for server, app/web site, and database is all in one place.

Regarding "discussions"...that is not really what I'm thinking of. More, moderated notes/tips? Not open and free form, it would be moderated and controlled, so only content that was really valuable would actually be displayed. The idea would be along the lines of, say, regarding the Triton kit: "Beware of the fittings, they tend to cause cracking in harder blank materials. Thinning the tube, or reducing the diameter of the part of each fitting that fits inside the tubes to a slip fit and gluing will avoid any risk to valuable blanks (i.e. TruStone)."

 

[jeff]

I am not experienced with either product, but a quick Google tells me both can be installed on Centos 7 so that's not an issue. If AWS is a better choice and IAP can own the AWS account, that would be good. (I've been meaning to try AWS for file storage and backups anyway...)

Regarding "discussions", I'm looking at it more as objective vs. subjective data. Maybe I'm old fashioned, but I've always felt that subjective data can unfairly taint objective data. In your example, who vets the opinion that the fittings crack harder blanks? For every person who agrees, there will be one who disagrees. Maybe a voting or moderation process handles that.

 

[Edgar]

I'm not sure I like the idea of making the source code open to the world. The database represents several man-years of work and is owned by Wayne & shared with the IAP community in pdf form & the Bushings & Tubes app. Further sharing through another app is one thing, but I'm not sure if we want to expose it any further than that.

 

[Edgar]

Ah-men to a unified location for instruction sheets.
One potential to save storage would be to just link to the existing vendor PDF of the instructions. You run the risk of having to retool if the vendor changes their naming format, but you are at least not on the hook for file storage/retrieval issues on both the front and back ends.


Possibly a two tier system - comments/reviews open to everyone with the ability to edit the core description to a dedicated group?

But yeah, I'd be interested in this as well. Count me in as a volunteer for just about anything short of the coding itself.

The biggest problem with linking to instruction files on vendor web sites is that most vendors delete those files when a kit is discontinued. Wayne & I have spent countless hours scouring Internet archives to accumulate copies of instructions to make them available via the Bushings & Tubes app. Storing them all in one place is not a big deal.

 

[Darios]

If AWS is a better choice and IAP can own the AWS account, that would be good. (I've been meaning to try AWS for file storage and backups anyway...)

One warning with AWS is it's byzantine and deliberately obscure billing practices, and it always (in my admittedly limited to one org's experience) is more expensive than it seems. It can take a fair amount of effort to put all the brakes in place to keep from blowing your budget. The Billing page claims to be able to put a caps on things but somehow things always slip around and/or you end up with the choice of having to cut services or pony up more cash to keep things up for the rest of the month.

edit - and with AWS you really, really want any outbound data to be as lightweight as possible.

 

[Darios]

I'm not sure I like the idea of making the source code open to the world. The database represents several man-years of work and is owned by Wayne & shared with the IAP community in pdf form & the Bushings & Tubes app. Further sharing through another app is one thing, but I'm not sure if we want to expose it any further than that.

Well, open source has a couple of big advantages, especially in volunteer built applications. Of lesser importance (IMO) in this scenario is the auditability of the code. Multiple eyes on it looking for flaws, weaknesses, et al helps make for stronger and more secure code.

The second and more relevant advantage here is that of continuity. We are all going to die. Some of us may be lucky to retire beforehand as well. Having the code 'out there' at least allows for the chance that someone else will be able to carry on and continue the work on it.

When you think about it - after having consolidated the files from elsewhere to build the database, the files are already 'out there'. You aren't really exposing much. Someone could duplicate your efforts, recruit their own volunteers to submit their data as well, or more likely (unless you've taken extraordinary efforts to lock it down - I don't have an apple product so can't know for certain) just pick apart the database that gets pushed with the app.

 

[Darios]

Maybe I'm old fashioned, but I've always felt that subjective data can unfairly taint objective data. In your example, who vets the opinion that the fittings crack harder blanks? For every person who agrees, there will be one who disagrees. Maybe a voting or moderation process handles that.

True. But https://www.penturners.org/threads/inferior-casting-stay-away.178090/#post-2189806 is a good example of a solid community discussion about a flaw - admittedly with a blank.

 

[wimkluck]

My advice is KISS. One or two people for the database. They get the input and moderation. They most valuable is the database so I would keep it on your own site.

 

[Edgar]

Well, open source has a couple of big advantages, especially in volunteer built applications. Of lesser importance (IMO) in this scenario is the auditability of the code. Multiple eyes on it looking for flaws, weaknesses, et al helps make for stronger and more secure code.

The second and more relevant advantage here is that of continuity. We are all going to die. Some of us may be lucky to retire beforehand as well. Having the code 'out there' at least allows for the chance that someone else will be able to carry on and continue the work on it.

When you think about it - after having consolidated the files from elsewhere to build the database, the files are already 'out there'. You aren't really exposing much. Someone could duplicate your efforts, recruit their own volunteers to submit their data as well, or more likely (unless you've taken extraordinary efforts to lock it down - I don't have an apple product so can't know for certain) just pick apart the database that gets pushed with the app.

What you say is generally true, but it's a matter of protecting intellectual property rights. We have taken steps to ensure that the database and apps will be preserved as long as IAP is around, even if us individual developers are not. IAP management has the discretion of deciding who will be allowed to continue that work in the future - it isn't open to the general public at this time.

While the data is certainly publicly available in pdf form or via app displays, the underlying data is protected as best we can. For example, the data is no longer distributed in spreadsheet form for that very reason. It was for a time, but we found that some people from other forums and groups were using and redistributing that data as their own without even crediting IAP as the source of their information. While making the data available in pdf form doesn't completely prevent that sort of thing, it makes it much more difficult, and whatever procedures they use to extract the underlying have to be repeated whenever we update our files if they want to keep their information current.

Of course we can't prevent anyone from doing the same things we did, nor would we want to. The raw data is readily available, but there is a LOT of work involved in extracting that data from web sites, instructions sheets, and other sources to compile it into a useful form. If anyone else wants to make that kind of time investment, more power to them. However, it says a lot that in the last 15 years, only one small group of volunteers has attempted to do so, and it was done primarily for the benefit of IAP and its members.

 

[Darios]

... it's a matter of protecting intellectual property rights...
... the underlying data is protected as best we can...
... it makes it much more difficult...

... only one small group of volunteers has attempted to do so...

My last 0.02¢ on this - but if might be worth a discussion that possibly the problem listed in that last line has something to do with the methodologies and mindset as defined by the first 3?

 

[Edgar]

I know that IAP has their bushings and tubes app, however support for Android has apparently been abandoned. Further, it appears as though the app requires a database download, so updates are slow and can require a hefty data requirement.

I'd just like to correct a couple of points about your initial post. It does take 30 minutes or more for the app to download the initial database when the app is first installed. However, that is a one-time thing and updates to the database only take a few seconds. Data storage for the app is about 2GB - not bad for info on over 4,000 kits with over 2,000 instruction files, and over 1,700 photos that are instantly available even without an Internet connection once it's been downloaded.

Android has not been abandoned. Unfortunately we don't have anyone willing to update it at this time. The source code and information needed to update it is readily available when and if we find someone who is willing to spend a little time to update the app.

 

[Edgar]

My last 0.02¢ on this - but if might be worth a discussion that possibly the problem listed in that last line has something to do with the methodologies and mindset as defined by the first 3?

Not really - the fact of the matter is that researching and extracting all the kit data required to build a database is a tremendously time-consuming activity, not to mention keeping it up to date. Wayne started this effort about 15 years ago and has repeatedly asked for assistance, but received very little. Initially, the data was made available in spreadsheet form (after all, it is compiled in simple csv files). While that is certainly the most user-friendly way to use the data, it created a number of problems from non-IAP folks using and redistributing the data as if it were their own. That led to protecting the raw data as best we could and sharing it through PDF files and the apps.

The matter is certainly open to discussion, but the compiled data IS intellectual property, and is deserving of some degree of protection and compensation for its use.

 

[Darios]

...and compensation for its use.

Ah. I had to check but that changes the scenario then.


IAP Bushings & Tubes Reference
penturners.org LLC
Designed for iPad

• USD 0.99

 

[Curly]

Ah. I had to check but that changes the scenario then.


IAP Bushings & Tubes Reference
penturners.org LLC
Designed for iPad

• USD 0.99

I'm pretty sure that 99¢ goes into the IAP to help with the hosting cost rather than Wayne, Edgar etc lining their pockets. A pretty small amount compared to other apps out there considering the usefulness to penturners .

My apologies if I read something into your post that isn't there.

 

[Edgar]

I'm pretty sure that 99¢ goes into the IAP to help with the hosting cost rather than Wayne, Edgar etc lining their pockets. A pretty small amount compared to other apps out there considering the usefulness to penturners .

My apologies if I read something into your post that isn't there.

That is correct - All proceeds (after Apple takes their cut) go to the IAP to help cover hosting costs.

To date, about 3,500 copies of the Apple version have been sold, generating a little over $2,000 in revenue for IAP over the past 7 years. $300 per year is not a lot, but it helps Jeff to pay the IAP bills.

Wayne & I donate all of our time, which represents several man-years of effort. We enjoy doing this, but we are also protective of the intellectual property that we have developed.

 

[Darios]

I'm pretty sure that 99¢ goes into the IAP to help with the hosting cost rather than Wayne, Edgar etc lining their pockets. A pretty small amount compared to other apps out there considering the usefulness to penturners .

Agreed, and that's what I figured. Just learning that they are charging for the data (remember - no apple product and never will have one) helped me understand why the pushback against more current development trends.

 

[jrista]

I'm not sure I like the idea of making the source code open to the world. The database represents several man-years of work and is owned by Wayne & shared with the IAP community in pdf form & the Bushings & Tubes app. Further sharing through another app is one thing, but I'm not sure if we want to expose it any further than that.

The DB data would not be included, only its schema.

 

[jrista]

I'd just like to correct a couple of points about your initial post. It does take 30 minutes or more for the app to download the initial database when the app is first installed. However, that is a one-time thing and updates to the database only take a few seconds. Data storage for the app is about 2GB - not bad for info on over 4,000 kits with over 2,000 instruction files, and over 1,700 photos that are instantly available even without an Internet connection once it's been downloaded.

Android has not been abandoned. Unfortunately we don't have anyone willing to update it at this time. The source code and information needed to update it is readily available when and if we find someone who is willing to spend a little time to update the app.

My goal is to create a unified app that works as phone apps, or as a web app, or even as an installable desktop app (we could bundle it say as an Electron app.) Single codebase, multiple deploy points. We can support caching, even preemptive, to support an offline mode...but, 2GB of data is a hefty load, and not necessarily necessary. I've built many apps that have a favoriting paradigm that caches copies of the data for favorited items so they are always available offline. That kind of supports a progressive download, but only of the things you are most interested in. Other data would still require a connection of some kind.

Longer term, it probably wouldn't be too hard to support a "cache it all" option, where those who have spotty connections can just download everything once and have it all.

 

[jrista]

What you say is generally true, but it's a matter of protecting intellectual property rights. We have taken steps to ensure that the database and apps will be preserved as long as IAP is around, even if us individual developers are not. IAP management has the discretion of deciding who will be allowed to continue that work in the future - it isn't open to the general public at this time.

While the data is certainly publicly available in pdf form or via app displays, the underlying data is protected as best we can. For example, the data is no longer distributed in spreadsheet form for that very reason. It was for a time, but we found that some people from other forums and groups were using and redistributing that data as their own without even crediting IAP as the source of their information. While making the data available in pdf form doesn't completely prevent that sort of thing, it makes it much more difficult, and whatever procedures they use to extract the underlying have to be repeated whenever we update our files if they want to keep their information current.

Of course we can't prevent anyone from doing the same things we did, nor would we want to. The raw data is readily available, but there is a LOT of work involved in extracting that data from web sites, instructions sheets, and other sources to compile it into a useful form. If anyone else wants to make that kind of time investment, more power to them. However, it says a lot that in the last 15 years, only one small group of volunteers has attempted to do so, and it was done primarily for the benefit of IAP and its members.

So, to be perfectly honest, this isn't quite what I'm looking to do, write an app for IAP... If this is the cost, that I'm completely controlled by IAP with regards to what I do here, then I may have to consider that... This thread has suddenly turned into something closer to "You'll do it our way or the highway"...and that kind of concerns me.

The data you guys have is truly valuable, and it would certainly save me (and potentially a community) a lot of time and effort, and I recognize that, and would greatly appreciate it. But...I don't want to suddenly find myself in a situation where what I can do or how is dictated entirely by a few people at IAP.

FWIW, when I started this thread, what I had in mind wasn't an "IAP" app... I've been planning to build myself an app, since an app from any other source that contains the information I need, simply wasn't available. I was just looking to see if a pen information app with support for community involvement was something people would be interested in. This thread has taken a rather stark turn over the day here... I really need to consider if joining forces here, is actually what I want todo... There are certainly benefits, I need to consider if the costs are worth a loss of control and flexibility with my own ideas.

 

[jrista]

I am not experienced with either product, but a quick Google tells me both can be installed on Centos 7 so that's not an issue. If AWS is a better choice and IAP can own the AWS account, that would be good. (I've been meaning to try AWS for file storage and backups anyway...)

Regarding "discussions", I'm looking at it more as objective vs. subjective data. Maybe I'm old fashioned, but I've always felt that subjective data can unfairly taint objective data. In your example, who vets the opinion that the fittings crack harder blanks? For every person who agrees, there will be one who disagrees. Maybe a voting or moderation process handles that.

So, I guess I'd like to hear your opinion here. Edgar has made a lot of comments here that concern me. When I posted this thread, what I had in mind was not exactly an "IAP app for IAP" really. I was planning to write an app, and wondered if the community here, had any interest in support for community involvement in building out a database of information (not necessarily JUST bushings and tube and other measurements) on pen making, and not necessarily just for kitted pens either.

This thread has taken a different turn, and, I would like to know where you stand. I can see that there would certainly be benefits for joining forces with you guys, but, at the same time, it is starting to appear that joining forces means giving up a lot of control over the design of the app, how it works, and there appears to be consternation over this data. I am honestly not sure if the benefits outweigh the costs to me and my original ideas now... Are you on the same page as Edgar here? How much control are you guys looking to have, if we join forces?

 

[jrista]

Well, open source has a couple of big advantages, especially in volunteer built applications. Of lesser importance (IMO) in this scenario is the auditability of the code. Multiple eyes on it looking for flaws, weaknesses, et al helps make for stronger and more secure code.

The second and more relevant advantage here is that of continuity. We are all going to die. Some of us may be lucky to retire beforehand as well. Having the code 'out there' at least allows for the chance that someone else will be able to carry on and continue the work on it.

When you think about it - after having consolidated the files from elsewhere to build the database, the files are already 'out there'. You aren't really exposing much. Someone could duplicate your efforts, recruit their own volunteers to submit their data as well, or more likely (unless you've taken extraordinary efforts to lock it down - I don't have an apple product so can't know for certain) just pick apart the database that gets pushed with the app.

Aye, the information and data can all be recreated...IF absolutely necessary. Hopefully we can figure something out here with regards to the IAP database...sounds like it is pretty thorough and accurate. The original idea I shared was a community-supported app that presented community-supported information and knowledge (not just code, but information and moderated, vetted knowledge) for pen making in general. With a community behind it, the data and files (including say uploading of PDF files for instructions, which could be done through the web site or app) could happen fairly quickly, as it wouldn't necessarily have to be done just by one or even a few people over years.

Open sourcing the code is, IMO, one of the best ways to ensure the longevity of a product. The situation right now is that the IAP Android app sadly has no support, so its stagnated...ironically, the very problem that has lead me personally to post this thread. An OSS app, one that was truly OSS with more than one champion and many contributors, should have an extremely low risk of something like that happening. That is the idea anyway. If there is concern about the database IAP has developed here, and keeping it under lock and key, well, then that is something to consider. I would really love to be able to use it, but if it comes with too many strings attached, it might not be that hard to build a new database of the same information (and potentially more).

FWIW, one thing I do not have in mind is making the app another discussion forum. IAP is much better suited to that, and I have no interest in competing there. I am interested in having a broadly deployable, broadly accessible, consistent source of information for....well, all useful details, instructions and concrete knowledge on pen making. Information and knowledge.

 

[jrista]

Not really - the fact of the matter is that researching and extracting all the kit data required to build a database is a tremendously time-consuming activity, not to mention keeping it up to date. Wayne started this effort about 15 years ago and has repeatedly asked for assistance, but received very little. Initially, the data was made available in spreadsheet form (after all, it is compiled in simple csv files). While that is certainly the most user-friendly way to use the data, it created a number of problems from non-IAP folks using and redistributing the data as if it were their own. That led to protecting the raw data as best we could and sharing it through PDF files and the apps.

The matter is certainly open to discussion, but the compiled data IS intellectual property, and is deserving of some degree of protection and compensation for its use.

This could still be done. There would be no need to publicize the data itself. When I mentioned open source and including "DB" stuff in the repo, I was thinking about database schema, the structure and relationships among tables (i.e. as .SQL files with the necessary scripting). The data itself, however, if there is a desire to keep that protected, then that shouldn't be a problem. A deployed server app would then have environmentally secured access credentials to the DB, so not only would access TO the database be protected, the raw data itself in its bulk form, would never in fact ever have to be distributed.

That simple fact should actually provide greater security to this admittedly precious resource you guys have compiled, as the DB wouldn't need to be transferred over the internet and stored on every user's phone (which potentially exposes it, to anyone who has enough knowledge to get it off their phones).

With the original idea I had of only keeping on people's devices the data they had favorited, only those bits of information would be stored on each user's device. There wouldn't be a need to download ALL of the data. Further, if it was absolutely necessary, we could introduce layers of security to thwart any attempts to directly access that data on said devices, if it was truly necessary. (Of course, whatever is displayed on a user's screen would be free for the taking, if someone really wanted to...but I gather its not the individual bits of data but the whole entire compilation, that is primarily of concern here.)

I have no problem keeping the data IAP has gathered protected, if that is the key requirement of its use.

 

[jeff]

So, I guess I'd like to hear your opinion here. Edgar has made a lot of comments here that concern me. When I posted this thread, what I had in mind was not exactly an "IAP app for IAP" really. I was planning to write an app, and wondered if the community here, had any interest in support for community involvement in building out a database of information (not necessarily JUST bushings and tube and other measurements) on pen making, and not necessarily just for kitted pens either.

This thread has taken a different turn, and, I would like to know where you stand. I can see that there would certainly be benefits for joining forces with you guys, but, at the same time, it is starting to appear that joining forces means giving up a lot of control over the design of the app, how it works, and there appears to be consternation over this data. I am honestly not sure if the benefits outweigh the costs to me and my original ideas now... Are you on the same page as Edgar here? How much control are you guys looking to have, if we join forces?

I have some thoughts on this. Edgar, Wayne, and I are discussing and I'll post when I have something of value to add.

However, going back to your OP, our position is not important to answering your question of interest from the community. This discussion has sort of veered off to discussing something with IAP involvement using the IAP/Wayne data, which isn't your goal.

The reason I initially mentioned IAP hosting, etc., is that I don't believe Wayne is going to provide the raw data without IAP involvement. It's his product and certainly his option, but as long as he's enjoying maintaining it, I don't know why he'd just give it up.

 

[jrista]

I have some thoughts on this. Edgar, Wayne, and I are discussing and I'll post when I have something of value to add.

However, going back to your OP, our position is not important to answering your question of interest from the community. This discussion has sort of veered off to discussing something with IAP involvement using the IAP/Wayne data, which isn't your goal.

The reason I initially mentioned IAP hosting, etc., is that I don't believe Wayne is going to provide the raw data without IAP involvement. It's his product and certainly his option, but as long as he's enjoying maintaining it, I don't know why he'd just give it up.

Ok, understood. I thought there was an offer to share the data, I did not know its history or the concerns around it.

If we can find a way to work it out, I'm happy to try. It is not an issue for me to try and keep the data secured. I just want to be able to maintain enough freedom to explore some ideas with the app, and it started to feel like if I used the data, then I'd be entirely under IAP control as far as what the app could and could not do.

I'm also happy to host however you guys prefer, if you want to take on the hosting. AWS is not a requirement. AWS has its conveniences, for sure, but if you guys are already paying for a server, then I don't see any reason to ADD COST to what you are already paying for. What I had in mind is pretty simple. It would be a Nest.js server, which is really just a fancy TypeScript wrapper around Node.js, which is super easy to host. The app itself would also be TypeScript.

We could technically use any DB server software for the database. PostgreSQL is probably the easiest for me to integrate with, as I know it quite well, use if often, and it is also free, open source, and again easy to install and host. I'm a bit averse to MySQL...despite its popularity, it has an endless multitude of problems and quirks that just make it hard to work with. I'm also well versed in MongoDB, if you wanted to go down that route, although it might not be the most appropriate solution. I also have over a decade of experience with Microsoft SQL Server, as well as Azure SQL, if you guys wanted to use that (hosting in the Azure cloud is easy...but again, would incurr additional cost).

PostgreSQL is probably the best option all around, as it should be no cost, no reason it couldn't be posted on your existing server(s), and would integrate seamlessly with the kind of Nest.js backend I'm planning on implementing.

 

[jrista]

@jeff Regarding cost, and covering cost...

I am also not averse to having a small fee associated with the app...or rather its usage. I know Apple app store rules might make this a bit of a mess, but maybe something like a yearly fee? Hosting is not free, bandwidth is not free. The fee doesn't need to be a lot, the goal is not to make a profit, just cover costs. An app fee is a one-time thing, which sadly, after a while, you lose revenue to cover costs, which is why I mention a subscription.

Let me know what you guys are thinking. I certainly don't want this to cost you guys more to host it, at least not without covering that cost.

 

[jeff]

This thread has taken a different turn, and, I would like to know where you stand. I can see that there would certainly be benefits for joining forces with you guys, but, at the same time, it is starting to appear that joining forces means giving up a lot of control over the design of the app, how it works, and there appears to be consternation over this data. I am honestly not sure if the benefits outweigh the costs to me and my original ideas now... Are you on the same page as Edgar here? How much control are you guys looking to have, if we join forces?

OK, sorry to take so long with a response. These are just basic thoughts...

You need the data in an SQL DB. We could easily create a PostgreSQL DB from our data. Once we do that, there are two data repositories being updated, and consequently no single, authoritative source. It would be a nightmare trying to propagate changes back and forth. All kinds of conflicts would arise that would require a human decision to resolve. Not feasible.

We could generate the files we need periodically from the Postgres DB. But then we have the same problem you identified above - giving up control of the data. For us, 'joining forces' means giving up control over the quality of the data.

For many years we've focused on providing a painstakingly curated source of data for practically every pen kit on the planet. While we want to make that data widely and conveniently available, we want to continue to maintain that data ourselves. I think it might be possible for us to work together in such a way that you don't give up control over the design of the app, and we don't give up control over the data.

 

[Jans husband]

All this makes me glad I don't know anything about IT. I have a son in law for that.
But it gives me the opportunity to give a big thank you to Jeff, Edgar, Wayne and everyone else involved in that side of things on our behalf.
Mike

 

[jeff]

I should add that we are painfully aware of the lack of Android support right now. This discussion was a much needed and appreciated nudge that we need to get our Android app updated and available. We just contracted with an Android app developer to help us move forward on this. I hope we are days away from success, but I will update when we know more.

 

[jrista]

OK, sorry to take so long with a response. These are just basic thoughts...

You need the data in an SQL DB. We could easily create a PostgreSQL DB from our data. Once we do that, there are two data repositories being updated, and consequently no single, authoritative source. It would be a nightmare trying to propagate changes back and forth. All kinds of conflicts would arise that would require a human decision to resolve. Not feasible.

We could generate the files we need periodically from the Postgres DB. But then we have the same problem you identified above - giving up control of the data. For us, 'joining forces' means giving up control over the quality of the data.

For many years we've focused on providing a painstakingly curated source of data for practically every pen kit on the planet. While we want to make that data widely and conveniently available, we want to continue to maintain that data ourselves. I think it might be possible for us to work together in such a way that you don't give up control over the design of the app, and we don't give up control over the data.

Thanks for the reply, Jeff.

I think we could make this work for both of us. In fact, if we do this right, it might help you guys manage your data better as well, make it easier for you guys to manage it.

Here is my thought. If we figure out an appropriate schema that suits your guys needs, as far as managing and maintaining your data... We can put that in a Postgres DB, and those tables would be under your control. I could READ that data, but not update it. That would keep you guys in control.

If we then create other tables for the data my app might generate. I could then write anything I needed for my app, any community involvement, to this other set of tables. I could ASSOCIATE this new, additional data to your original data in your tables, via database keys. Every record in a DB would have a key of some kind. Those keys shouldn't change once a record is created for a given entity, so I should be able to reference them. I might be able to, say, associate a recommended correction, to some particular pen kit, say its bushings specs. This recommendation would be SEPARATE and distinct from the original data. I could show both the original, and the recommended correction, in my app (and probably in various ways, I could even consolidate results so that corrections override your data only once all the information has been read out of the DB, for example). I could even allow users to traverse a history of "changes", by treating the original data from you guys as the historical "basis", then progressively applying corrections on top of that, dynamically. I could also potentially allow people to add new kits (or even other things besides just pen kits, such as materials, tools, maybe design plans, etc. required to create a custom kitless pen design, etc.) by storing that data in my own tables, separate (but maybe identical in schema) to yours.

Further, all this information would be in the same DB. You guys could choose to leverage any additional data I create, or not, as part of the curation process for your data. Would entirely be up to you. But, it could be helpful in giving you guys an additional resource to identify discrepancies in the data. You could choose to include new kits the community of users of my app add, or not, to your curated data. That would all entirely be up to you guys.

You guys could also generate any other representation of the data you are curating from the DB. You could query the tables to generate CSV output, then use that to drive the generation of your PDFs, etc. I think there is a huge amount of potential around putting your curated data in a DB, regardless, even if you don't share it with me. Once it is normalized and stored in relational tables, you guys should gain a lot of power. Further, if you keep a running set of backups of the database (normally, you would want something like weekly full backups, then daily differential backups), you could also have that historical record to reference, in case any mistakes were made during curation. Store your backups in a few geographically diverse locations (i.e. you could keep copies on your guys home computers, or store them in a couple of AWS S3 buckets even), and the data should survive localized catastrophes.

Actually, another way to maintain a history for your data, is to set up history tables. With Postgres, you can set up "triggers", such that any time a record in one table is added or updated, you can first create a copy of the current state of teh record in a companion "history" table. This can be done easily enough with an BEFORE trigger. You can also have AFTER and INSTEAD OF triggers. If you primarily maintained your curated data through the DB, then it could be set up to automatically maintain a history of all changes (prior versions of each record, basically) for you, which might be quite valuable.

Anyway. Once we have a database with the data, a lot of possibilities open up. Again, I have no problem accommodating your need to protect the data. I'm happy to use it on a read-only basis, and then only write to my own distinct set of tables. With DB permissions, you guys should be able to restrict what tables I can write to, so that my app could literally only have "read access" to your curated data tables, to guarantee that data is safe.

Thoughts?

 

[jeff]

@jrista

Some good thoughts in there. I still see the potential for divergence of the data, but describing the scenarios here probably isn't necessary.

I definitely see the benefit of putting our data into a DB. It's been on my plate for some time, waiting for "some time"

Another scenario might be to maintain our data and your data in completely separate DBs. We could host ours, and you could host yours. You could pull the underlying 'IAP Reference Data' from ours, and display ours or yours depending on whether changes exist for that piece of data.

If you have some time next week, maybe we could have a phone call.

 

[jrista]

@jrista

Some good thoughts in there. I still see the potential for divergence of the data, but describing the scenarios here probably isn't necessary.

I definitely see the benefit of putting our data into a DB. It's been on my plate for some time, waiting for "some time"

Another scenario might be to maintain our data and your data in completely separate DBs. We could host ours, and you could host yours. You could pull the underlying 'IAP Reference Data' from ours, and display ours or yours depending on whether changes exist for that piece of data.

If you have some time next week, maybe we could have a phone call.

Hi Jeff,

I think we could put together a call. I'd like to hear your concerns. I think if we manage permissions properly, it'll prevent any chance of any of my code from messing with your data. There are distinct performance and bandwidth benefits from having all the data in a single db (i.e. direct joins, which can be done in the DB itself, reducing (often greatly) the amount of data that needs to be transferred to my server code). I am planning on using an ORM, which can do other optimizations in-db, while maintaining a nice programmatic object model for my source code to work with.

I'm pretty sure I can alleviate any fears you have of the data being modified or diluted by my app. We could use db schemas as well as a means of segregating data and managing access. Postgres is a very powerful DB. I'm most familiar with SQL Server, which is extremely powerful, and Postgres comes in a very close second (with some unique capabilities of its own.) I think with this database server, you'll find you have immense control.

 

[jrista]

@jeff Hey, sorry for letting this fall through the cracks. I've had some things come up that I had to deal with, as I needed clear, warm, dry weather to do so. Also ran up against a deadline at work.

I think, once the holiday is past here, I should have time to get back to thinking about this. Would still like to chat about the options, see what we can do.