I've got all the parsing and massaging from Slack, but can't post to Discourse

After much packing, etc., I had some time tonight to thwack at the migration problem. I think I’ve finished all the parsing and massaging of the Slack data, and was actually all excited to try to post a few test messages to Discourse from the 1,400+ Slack messages.

But all my attempts to create either a topic or a post (message) using the Ruby gem fail. I know my key, etc., works, because I can connect and download things like recent posts and a list of users. When I try to create something, I get back a list of the 50 most recent posts, with no error info that would help me understand what’s up. I’ve posted a question on the Discourse site and am going to bed.

Hopefully, though, it’s a simple little thing, and we’re really close to migrating our Slack data.

Still no progress on figuring out why I can’t post. The Discourse people seem to think that is should work, so things are sort of stalled there.

Can someone on the Hampshire admin side (perhaps @admin or @vgabruzzo ?) have a look at the logs and see if there’s anything there that would help us understand what’s dying?

Some things I’ve tried and the approximate times I tried them:

  • 28 July, 1am, I tried to create a new category called “Test category” via the API.
  • Very late 26 July into very early 27 July, I tried creating topics and (I think) posts in the Slack Archive category.

All actions would have been as slack_archive user.

I’d be happy to try to create things again at a specific time if that would make it easier to find things in the logs.

It would be cool to know if there’s anything in the logs around those times that might help us debug this.

Thanks!

Hi,

If you log into the slack_archive account, you should be able to access the error logs via the admin panel. There are 4 ‘invalid_access’ errors near the end that were caused by the archive account. You can view the info, backtrace and env for each error. It seems that each error has an invalid username/pw combination. For some reason there is an appended “_” or “xxx” on either the username or pw.

That is a good place to start. But, if you’ve tested the script more than 4 times, I’m guessing the script sometimes doesn’t throw an error in discourse itself when the un/pw are both correct. I’ll dig through some logs on the server. In the meantime can you post the code here or in a gist?

I was able to create a test topic using the JSON API via cURL rather than the Ruby gem as per: https://meta.discourse.org/t/discourse-api-documentation/22706

The Ruby gem is just a wrapper around this, so I’m pretty sure the server is set up to correctly handle this. One thing that comes to mind is that you should definitely us https rather than http. Other than that I’d have to look at the Ruby. As a last resort, you can probably write a Bash script to do what you want, since the API seems to work fine when using curl.

Those were specific tests on my part to confirm that I could break the authentication if I changed the username or password. If those were the only such errors, then that suggests that the other hits didn’t leave something in the logs.

That does now seem the likely culprit. I’m about to test it and see, and shall report back shortly!

The problem was indeed that I was using http instead of https. Grrrr.

Huge thanks to @vgabruzzo for helping with this!

@vgabruzzo - I suggested to the Discourse people that returning the 50 most recent messages wasn’t a great “error” message. Their response was that the problem was actually our http to https forwarding. Does that make sense? Is there something we should look at on our end?

Thanks!

Yeah that makes sense. We force https, so what was happening should have been happening. However, I wasn’t getting the last 50 latest topics as my response when using http; I was getting an actual redirect message.

As to why you were getting the latest 50 topics and not the redirect message, I’m not sure–I’d have to take a look at the Ruby code.

I’m glad you got it working! Once it’s polished up it might be something worth sharing with the open source community.

-Vince

Sharing back had been my original plan, but I ended up cutting a lot of corners and ignoring a lot of cases that we didn’t need just to get the thing to move forward. So while I really like the idea, there would have to be a lot of caveats at the moment.

When I do the final migration I’ll also get the latest code up on Github so we can talk about the specific issues that would need to be addressed before it might be genuinely useful to someone else.