You are here

https problem

22 posts / 0 new
Last post
seitan
Offline
Joined: 10 months
Last seen: 10 months
https problem

Hello forums,

I am unable to grab EPG from TV program site, since it moved from http to https. Webgrab+ throws error:

Error downloading page: Error getting response stream (Write: The authentication or decryption has failed.): SendFailure

Site is https://www.tv24.lt

I use mono tu run Webgrab+.

Mono version:

Mono JIT compiler version 3.2.8 (Debian 3.2.8+dfsg-10)

Is there any chance to solve this problem? Thank you.

 

 

lawin1
Offline
Donator
Joined: 1 year
Last seen: 1 month

http://www.webgrabplus.com/content/https-and-mono

It's a known problem. We can only hope that Mono will be updated soon. (Man taip pat nuo šios savaitės nepavyksta atnaujinti https://www.tv24.lt  EPG su Raspberry Pi 2, kuriame instaliuota naujausia Ubuntu Mate 16.04 ir Mono JIT compiler version 4.2.1 (Debian 4.2.1.102+dfsg2-7ubuntu4)).

Blackbear199
Offline
Joined: 1 year
Last seen: 8 min

the problem is with mono.it has issues with certain types of encryptions.this site uses one of them.

there is a workaround thanks to becks14 i'vew been testing it and seems to work fine.

fisrt you need to install a server and php,it wont work without it.lots of tutorials on the net for how to install this on your linux distrubition.

that said,howto install

copy the attatched php script to your server.rename it from tv24_lt.php.txt to tv24_lt.php

included is a updated ini file.

how it works

open the ini file and near the top you will see 2 url_index lines,one labeled for windows,one for linux.since were talking linux here its enabled already(same method is used for channels.xml creation if you need create a new one).

for the linux url_index you will see it starts with(http changed so it will show correct in forum) hxxp://127.0.0.1/tv24_lt.php

127.0.0.1 is your server ip,i run nginx on same pc as webgrab so thats what i use.

tv24_lt.php is the name of out script.

so as you can see that rather than connect to the site directly we are calling our php script.

in our php script you will see a bunch of stuff(if you know php then you can see its not very complicated) so i'm just gonna explain whats important for someone that dont.

// epg grabbing enable only 1 $url line below // disables line

$url = "https://www.tv24.lt/programme/listing/none/" . $urldate . "?filter=channel&subslug=" . $channel;

// channels.xml creation

//$url = "https://www.tv24.lt/data/channels/visi";

first in php double foreward slashes(//) disables a line when in webgrab ini files we use the *

its self explainatory,first $url is for epg grabbing and second is for channels.xml creation

enable only 1,by default epg grabbing is enabled so if you need the create a new channels.xml then you will need to disable it and enable the channels.xml,change it back when done.

so  all were doing here is using a server to host our php script.

we changed our url_index in the ini file to call it.

this gets us out of the mono environment and the https problem.

the php script handles the https session and sends the reply's back to webgrab.

 

 

 

 

 

lawin1
Offline
Donator
Joined: 1 year
Last seen: 1 month

Installed apache and php:

sudo apt-get install apache2 -y

sudo apt-get install php libapache2-mod-php php-curl php5-gd -y

 

I can access default Apache html file and I have tested php with an info.php (<?php phpinfo(); ?>) in the /var/www/html. At first I was getting 500 Internal server error, but after Raspberry Pi reboot Webgrab stops with this error:

file /home/pi2/wg++/guide.xml not found, creating a new one ..

update requested for - 97 - out of - 97 - channels for 8 day(s)
update mode - full - for all channels

      i=index  .=same  c=change  g=gab  r=replace  n=new

channel (xmltv_id=LRT Televizija) site -- TV24.LT -- mode full
iiiiiiiitime parsing error : String was not recognized as a valid DateTime.
stoptime time scrubbed : 1463866500
computer date/time format: 5/22/2016 7:17:44 PM

 

 

Blackbear199
Offline
Joined: 1 year
Last seen: 8 min

upload your log file

 

Blackbear199
Offline
Joined: 1 year
Last seen: 8 min

also use latest webgrab beta

 

lawin1
Offline
Donator
Joined: 1 year
Last seen: 1 month

Log.

Attachments: 
Blackbear199
Offline
Joined: 1 year
Last seen: 8 min

looks to working,u just need latest webgrab beta.

also once it is working you can open your ini file and remove (debug) from here..

index_showsplit.scrub {multi(debug)||||} * copy content

so its

index_showsplit.scrub {multi||||} * copy content

i had added it when i tested the script.it dont hurt anything just creates a larger log file when the data is pulled.

 

lawin1
Offline
Donator
Joined: 1 year
Last seen: 1 month

Copied a line with an "url_index" from your newest tv24.lt_.ini to an initial tv24.lt.ini and removed original url_index{url|http://www.tv24.lt/programme/listing/none/|urldate|?filter=channel&subslug=|channel}. Finally it works with https! I will test latest webgrab beta later. Thank you very much Blackbear199.

meelis
Offline
Joined: 10 months
Last seen: 10 months

Thanks! Works also for kava.ee. Just replace tv24.lt with kava.ee.

francis
Offline
francis's picture
WG++ Team memberDonator
Joined: 4 years
Last seen: 1 week
Is the support helpful?
support us

This should now work out of the box (so no php needed)

see http://www.webgrabplus.com/content/why-https-fails-mono

Netuddki
Offline
Joined: 10 months
Last seen: 2 hours
francis wrote:

This should now work out of the box (so no php needed)

see http://www.webgrabplus.com/content/why-https-fails-mono

There is not even a mono 4.9.0 out yet

and I can't figure out how to install the latest 4.8.5 alpha version.
can somebody tell me?

Blackbear199
Offline
Joined: 1 year
Last seen: 8 min
Netuddki
Offline
Joined: 10 months
Last seen: 2 hours

I did all of that. Three times.

It's still 4.6.1.5 when I use "mono --version".

 

That site is a huge mess.
There is not one straight line, what to do.
I don't need blahblah.

I need this:
You idiot, execute command 1, command2, command 3.
And then it should work.
I don't care what it does or why it does that.

I am really pissed.

Blackbear199
Offline
Joined: 1 year
Last seen: 8 min

if you followed the steps in the first link correctly you should be able to use your package manager and install it.

as you can see if your using a ubuntu based distro the latest version is 4.8.0.374,theres a update available that i havnt installed yet.thats why mine is showing 4.8.0.371 and the box to the left is grey with the mark.

 

 

Netuddki
Offline
Joined: 10 months
Last seen: 2 hours

I did this in putty

echo "deb http://download.mono-project.com/repo/debian alpha main" | sudo tee /etc/apt/sources.list.d/mono-xamarin-alpha.list sudo apt-get update

 

And then what?

There is nothing described how to install it.

It's just blah blah.

I have no package manager or anything near like you have in the picture.

I have a black window and I can use command lines. That's it.

Please show me, what I have to do from this, because for me it's just a big pile of BS:

Usage

The package mono-devel should be installed to compile code.

The package mono-complete should be installed to install everything - this should cover most cases of “assembly not found” errors.

The package referenceassemblies-pcl should be installed for PCL compilation support - this will resolve most cases of “Framework not installed: .NETPortable” errors during software compilation.

The package ca-certificates-mono should be installed to get SSL certificates for HTTPS connections. Install this package if you run into trouble making HTTPS connections.

The module mono-xsp4 should be installed for running ASP.NET applications.

Notes

After the installation completed successfully, it’s a good idea to run through the basic hello world examples on this page to verify Mono is working correctly.

Mono on Linux before 3.12 by default didn’t trust any SSL certificates so you got errors when accessing HTTPS resources. This is not required anymore as 3.12 and later include a new tool that runs on package installation and syncs Mono’s certificate store with the system certificate store (on older versions you had to import Mozilla’s list of trusted certificates by running mozroots --import --sync). Some systems are configured in a way so that the necessary package isn’t pulled in when Mono is installed, in those cases make sure the ca-certificates-mono package is installed.

Accessing older releases

If for some reason you want to pin an older version of Mono rather than updating to the latest, you can modify the Debian repository to “wheezy/snapshots/X.XX.X” instead of “wheezy”. For example, “wheezy/snapshots/3.10.0” will lock you to that version.

These snapshots will cease receiving updates as soon as the next major Mono version gets uploaded - for example, as soon as Mono 3.12 gets uploaded, 3.10 will never receive updates.

On RPM distributions, force the package version in your package manager - all older versions are published in the YUM metadata and should be available.

Security updates for older releases

Due to our repository management software, we are unable to directly offer security fixes in our standard repositories for any version of Mono other than the latest.

If you are running a snapshot version, and want security support, add a repository as appropriate from the list below:

echo "deb http://download.mono-project.com/repo/debian 38-security main" | sudo tee -a /etc/apt/sources.list.d/mono-xamarin-security.list
echo "deb http://download.mono-project.com/repo/debian 310-security main" | sudo tee -a /etc/apt/sources.list.d/mono-xamarin-security.list
echo "deb http://download.mono-project.com/repo/debian 312-security main" | sudo tee -a /etc/apt/sources.list.d/mono-xamarin-security.list
echo "deb http://download.mono-project.com/repo/debian 40-security main" | sudo tee -a /etc/apt/sources.list.d/mono-xamarin-security.list

Alpha update channel

If you want to test bleeding-edge updates to Mono, we now have an alpha update channel. Please report bugs promptly, so we have time to fix them before packages are migrated to stable!

Debian, Ubuntu, and derivatives (alpha channel)

echo "deb http://download.mono-project.com/repo/debian alpha main" | sudo tee /etc/apt/sources.list.d/mono-xamarin-alpha.list
sudo apt-get update

CentOS, Fedora, and derivatives (alpha channel)

yum-config-manager --add-repo http://download.mono-project.com/repo/centos-alpha/

This repository URL is also valid for SUSE when added to YaST. This can be done using the YaST GUI, or via zypper:

zypper ar -f http://download.mono-project.com/repo/centos-alpha/ mono-project-alpha

Beta update channel

If you want to test bleeding-edge updates to Mono, we now have a beta update channel. Please report bugs promptly, so we have time to fix them before packages are migrated to stable!

Debian, Ubuntu, and derivatives (beta channel)

echo "deb http://download.mono-project.com/repo/debian beta main" | sudo tee /etc/apt/sources.list.d/mono-xamarin-beta.list
sudo apt-get update

CentOS, Fedora, and derivatives (beta channel)

yum-config-manager --add-repo http://download.mono-project.com/repo/centos-beta/

This repository URL is also valid for SUSE when added to YaST. This can be done using the YaST GUI, or via zypper:

zypper ar -f http://download.mono-project.com/repo/centos-beta/ mono-project-beta

 

 

Blackbear199
Offline
Joined: 1 year
Last seen: 8 min

there's no idiots guide to linux but i'm sure you could manage on google "how to install a package in terminal"

sudo apt-get install mono-complete

sudo apt-get install ca-certificates-mono

then do what francis says in his link,read the last 2 posts here about the last command  btls-cert-sync,it wasnt included in the alpha when i installed it.i uploaded in this post.

http://www.webgrabplus.com/content/https-and-mono

if your not willing to learn linux is not for you,theres tons of info out there on how to do stuff.do a simple search.

 

 

Netuddki
Offline
Joined: 10 months
Last seen: 2 hours

Thanks for your help!

It works now.

I can't learn linux without an idiot's guide.

No matter what I look for for linux, there is always just a bunch of commands. I learned to accept it.
But then when they suddenly assume, that I know everything and on a "How to install" page they say "yeah, install it" and they are not able to write down two friggen lines, like you did, then I think, I am not the one, who didn't do his "job".

francescoratta
Offline
Joined: 6 months
Last seen: 5 days

How is the procedure if using other ini files? Like for example tv.zam.it?

I have a few websites which are giving me this error, so I would like to correct the problem using your php method, as I have nginx server running with php on my Freenas (Freebsd)

Like I said I am running mono on a Freenas server (Freebsd) but it is recommended by the Mono project to stick with the versions included with freebsd ports, as they may include specific patches and stuff specifically for the freenas system.

So I tried to modify slightly the ini file and the php script just to see if I could make it work, but I get syntax errors that no index can be found.

Included is my two files.

 

Thank you in advance.

 

Blackbear199
Offline
Joined: 1 year
Last seen: 8 min

first thing you need to do is figure out is why its not working.

assuming or guessing doesnt work to well.you will end up going around in circles.

tvzam is not even using https so it not a mono/https problem.

if you look in the ini near the top and the last revion date you will see its was in 2014.

not to many ini make it that long without something changing on the site(some do though).

here's a updated ini

i cannot remember if the channels.xml is the same as the old one so either compare it to the included or use the included one.

 

 

 

francescoratta
Offline
Joined: 6 months
Last seen: 5 days

Hi,

The reason why it wasn't working was due to some channels from tv.zam.it were giving me the following error: "The authentication or decryption has failed".

Then searching through the forums I stumbled across this post, but as you correctly pointed out this website doesn't use HTTPS, which also confused me.

Anyways I am happy to know that there was a updated ini file, even though when I installed Webgrab v2 it downloaded automatically a updated site.ini pack...

Thank you for the quick answer!

 

francescoratta
Offline
Joined: 6 months
Last seen: 5 days

By the way, where is the best place to stay up to date with the ini files? I found the github page for the site.ini pack, but I noticed that those files are not up to date. This updated tv.zam file for example still has the old version in github and also if you download the updated pack using the program.

Would be nice if all updates goes to github instead of having to search through the forums for it.

 

Thank you again for the great work much appreciated.

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl