You are here

Updated package for DSMART, HURRIYET, TRT and DIGITURK

24 posts / 0 new
Last post
mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year
Updated package for DSMART, HURRIYET, TRT and DIGITURK

Hi All,

1 days ago I joined this forum and I tried to fix ini files for Turkish channels. Thanks to Blackbear199, I fixed all of them. I create special config file for best EPG coverage for all channels. I mean some of channels are in 2 or 3 ini files. I picked best of one. You should change channel name with your m3u file. I made an excel file to show which ini has the best EPG data for channels. Just DSMART uses php file. (it is in zip file)

I hope it will be useful for everyone. I found another EPG sources for tivibu channels. Maybe someone can create another ini for tivibu:

http://www.tivibuspor.com.tr/yayin-akisi

Another one for beIN Sports Turkey : http://tr.beinsports.com/yayin-akisi/beinsports

WebGrab

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

Hello,

You need a webhost for php file. If you are using Windows 7-8-10 you can use IIS Web Server it is Microsft's Built-in solution. You can install from Control Panel Add/remove Windows Feature. After finish you IIS installation put test.php file to C:\inetpub\wwwroot folder. Default IIS port is 80 I use 83 for this reason you need to change the line starts with 127.0.0.1:83 to 127.0.0.1:80 in ini file. That's it.

Alternative: You can find free php host on the internet. You can use free hosts but you should change the URL in ini file.

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

You're welcome wink

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year
Netuddki wrote:

You don't need php anymore, because mono was fixed on linux for https sites.

I don't want to burst your bubbles, but there are already better dsmart inis on the forum.
What you did, is a very simple thing and won't get you anywhere.

For example this one (which is still not perfect because sorting has errors, but I will fix it):

 

 

Hey kiddo, calm down. There is no competition in this website. You should change your attidude before you continue.. I checked all of ini files for dsmart none them was working properly on -WINDOWS- except Blackbear199 php file. What I did, I added some lines to php for generating channel list and I changed site_id. Yes this is not big changes I didn't launch a rocket for mars but I believe it was a good fix for Turkish users. I don't need your or someonelese's appreciations. Capish? Thanks to Blackbear199 ini file fixed and working properly for WINDOWS users.

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year
Blackbear199 wrote:

all plse calm down...

this site is not as simple as it seems..

it your using this(mkaand i think ur using the other i made)

url_index {url|https://www.dsmart.com.tr/actions/schedule?channel_id=|channel|&day=|urldate|}

this is a royal pita

the times are correct but the date is not in the "start_date": "xx"

study the data and see for your self...

i still have this ini open trying to figure out whats going on...

I didn't check all channels that I grap from DSMART but I pick FOXCrime Currently 2:15 AM in Turkey and MacGyver just started and looks OK. But I didn't check every channel one by one.

2017_08_25_02_17_14 2017_08_25_02_18_57

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

FX 2:25 Twin Peaks (2017) Looks OK!

screenshot000

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year
Blackbear199 wrote:

if its showing correct for you...great but i have no idea why..

add debug to your showsplit and loo at the "start_date": "xx"

what i amd seeing for today its like 3 days earlier(the date it shows with correct time),tomorrow is different again.

 

I think you're right and wrong. I mean the main core of code working (your code) working perfectly for some channels in DSMART. For example:

There are two A Spor channel in DSMART channel list one HD one SD EPG data (same channel). 

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c6062b2" xmltv_id="A Spor HD">A Spor HD</channel>

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c6062a3" xmltv_id="A Spor">A Spor</channel>

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c60627b" xmltv_id="24 Kitchen Hd">24 Kitchen Hd</channel>

 

A Spor channels returns NO DATA but 24 Kitchen returns correct data. Here is the link for debug if you want to check:

 

https://www.dsmart.com.tr/actions/schedule?channel_id=58d29bb0eefad3db9c...

https://www.dsmart.com.tr/actions/schedule?channel_id=58d29bb0eefad3db9c...

 

Actually source has data but code cannot grab. Maybe you need to add a poke-yoke. I have no idea. But still your PHP solution is the best solution for windows users. I need to check all channel one by one.

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c606243" xmltv_id="ULKE TV">ULKE TV</channel>
    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c60629f" xmltv_id="Ulke Tv HD">Ulke Tv HD</channel>

These two are interested also. First one returns NO DATA but second one returns DATA.

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

Yesi now ı understand what you say. But looks like this is not your fault. The source has wrong date for some channels. But I don't know main DSMART page:

https://www.dsmart.com.tr/yayin-akisi

is using same data or not but visit above side and find A Spor. It looks OK.

2017_08_27_01_57_18

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

I am not make sure but I think I understood how is working DSMART's API. Please visit this JSON viewer:

http://jsonviewer.stack.hu/#https://www.dsmart.com.tr/actions/schedule?c...

All the programs that shows (Entries) for today's program. Because our query has 2017-08-27

Start and End dates just ignored but the times are valid for GMT. In Turkey we use GMT+3 so everything is clear now:

2017_08_27_02_17_00

Currently broadcasting this one "Program Tekrarı" start date 23:00:00Z (GMT) main page shows 02:00 thats correct! The system ignores dates and uses times only I guess.

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

Can you modify latest DSMART.ini above with this logic? I mean just grab start and end time. I added Task Scheduler for my Windows 10. Every day 10 AM above config file runs. So it is not important to grab tomorrow and other days but of course if you can grab all of data it would be great. As I said before they use only time not date. But like 24 Kitchen channel sometimes day added correct day info also. That's why your code ini file working for some channels and not working for some channels I guess. Now time to go to bed 02:31 AM If you need help for testing just let me know wink

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

Voila!! I just replaced two lines and looks like it fixed. I was testing on this channel:

    <channel update="i" site="dsmart.com.tr" site_id="58d29bb0eefad3db9c606235" xmltv_id="FB TV">FB TV</channel>

And here is the result:

<?xml version="1.0" encoding="UTF-8"?>
<tv generator-info-name="WebGrab+Plus/w MDB &amp; REX Postprocess -- version  V2.1 -- Jan van Straaten" generator-info-url="http://www.webgrabplus.com">
  <channel id="FB TV">
    <display-name lang="tr">FB TV</display-name>
    <url>http://www.dsmart.com.tr</url>
  </channel>
  <programme start="20170826210000 +0000" stop="20170826220000 +0000" channel="FB TV">
    <title lang="tr">Haber</title>
  </programme>
  <programme start="20170827213000 +0000" stop="20170827220000 +0000" channel="FB TV">
    <title lang="tr">Haber</title>
  </programme>
  <programme start="20170827220000 +0000" stop="20170828063000 +0000" channel="FB TV">
    <title lang="tr">Fenerbahçe - Vardar</title>
  </programme>
  <programme start="20170828063000 +0000" stop="20170828073000 +0000" channel="FB TV">
    <title lang="tr">Günaydın Fenerbahçe</title>
  </programme>
  <programme start="20170828073000 +0000" stop="20170828083000 +0000" channel="FB TV">
    <title lang="tr">Haftalig</title>
  </programme>
  <programme start="20170829080000 +0000" stop="20170829083000 +0000" channel="FB TV">
    <title lang="tr">Sinema Sinema</title>
  </programme>
  <programme start="20170829083000 +0000" stop="20170829093000 +0000" channel="FB TV">
    <title lang="tr">Stüdyo Fenerium</title>
  </programme>
  <programme start="20170830090000 +0000" stop="20170830093000 +0000" channel="FB TV">
    <title lang="tr">Güncel</title>
  </programme>
  <programme start="20170830093000 +0000" stop="20170830100000 +0000" channel="FB TV">
    <title lang="tr">Sinema Sinema</title>
  </programme>
  <programme start="20170830100000 +0000" stop="20170830110000 +0000" channel="FB TV">
    <title lang="tr">Haber</title>
  </programme>
  <programme start="20170831104500 +0000" stop="20170831123000 +0000" channel="FB TV">
    <title lang="tr">Fenerbahçe - Gençlerbirliği</title>
  </programme>
  <programme start="20170901110000 +0000" stop="20170901123000 +0000" channel="FB TV">
    <title lang="tr">Zalgiris Kaunas - Fenerbahçe Ülker</title>
  </programme>
</tv>

But it stranges I can still see 26-08-2017. It's already passed but it is OK. Thank you very much. You grabbed all of data with just changing two lines. Problem solved I guess isn't it?

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

Maybe I should change UTC+3 I didn't check the times. But it grabs with this ini. Tested.

site {url=dsmart.com.tr|timezone=UTC|maxdays=7|cultureinfo=tr-TR|charset=UTF-8|titlematchfactor=90}
site {episodesystem=onscreen}
*
url_index{url(debug)|http://127.0.0.1:83/test.php?channel=|channel|&date=|urldate|&reqtype=1}
url_index.headers {customheader=Accept-Encoding=gzip,deflate}
urldate.format {datestring|yyyy-MM-dd}
*
index_showsplit.scrub {multi|"schedule": [{||}]}|}]}}

 

scope.range{(splitindex)|end}
index_showsplit.modify {replace|\}, \{|\|}
index_showsplit.modify {sort(ascending,string)}
sort_by.scrub {single(target="index_showsplit")|"start_date": "||",|",}
sort_by.modify {calculate(target="index_showsplit" format=date,unix)}
index_showsplit.modify {(debug)} *for debug
end_scope
*
index_start.scrub {single|"start_date": "|T|Z|"}
index_stop.scrub {single|"end_date": "|T|Z|"}
index_title.scrub {single|"program_name": "||",|",}
index_title.modify {cleanup(style=jsondecode)}
index_description.scrub {single|"description": "||",|",}
index_description.modify {cleanup(style=jsondecode)}
index_category.scrub {single|"genre": "||",|",}
index_category.modify {cleanup(style=jsondecode)}
*

**  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _  _
**      #####  CHANNEL FILE CREATION (only to create the xxx-channel.xml file)
**@auto_xml_channel_start
*url_index {url|http://127.0.0.1:83/test.php?reqtype=3}
*url_index.headers {customheader=Accept-Encoding=gzip,deflate}
*index_site_id.scrub {multi|"_id": ||,|,}
*index_site_id.modify {remove|"}
*index_site_id.modify {remove(notnull type=string)|'}'}
*index_site_id.modify {cleanup(removeduplicates=equal,100 link="index_site_channel")}
*index_site_id.modify {cleanup(style=jsondecode)}
*index_site_channel.scrub {multi|"channel_name": "||"|"}
*index_site_channel.modify {cleanup(style=unidecode)}
**@auto_xml_channel_end

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

I tested your latest modification above. I didn't add UTC+03:00 EPG data shows correct for spme channels (before modification these channels already working with UTC+00:00) If I add UTC+03:00 new EPG data shows on Kodi but all of them wrong. Looks like your fix is not valid fix for this web site. Because still dates are wrong on guide.xml. We need modification. It should grab only TIME not for date. 

For your other question. I have Raspberry Pi Zero. Can I run WebGrabPlus on raspian? It based on linux. But If I do that I need this feature:

After generation of guide.xml linux version should send this file to my FTP. Can I do that? Thanks.

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

I added this line to my config (Because I am trying to debug)

  <timespan>0</timespan>

Just I want to grab today's EPG data for UÇANKUŞ TV with your latest start end time modification:

What I grabbed:

  <programme start="20170827214500 +0000" stop="20170827220000 +0000" channel="Uçankuş TV">
    <title lang="tr">UçanKuş'ta Astroloji</title>
    <desc lang="tr">Dr. Astrolog Şenay Yangel, her gun burçları yorumluyor..(n)</desc>
  </programme>
  <programme start="20170827220000 +0000" stop="20170827224500 +0000" channel="Uçankuş TV">
    <title lang="tr">Maraba Televole</title>
  </programme>
  <programme start="20170827224500 +0000" stop="20170827233000 +0000" channel="Uçankuş TV">
    <title lang="tr">İstanbul Geceleri</title>
  </programme>
  <programme start="20170827233000 +0000" stop="20170828001500 +0000" channel="Uçankuş TV">
    <title lang="tr">Kuş Yuvası</title>
    <desc lang="tr">Enerji ve renk uzmanı Sebla Öztürk, ünlü konuklarıyla daha enerjik, daha rahat bir yaşam için neler yapılması gerektiğini konuşuyor(n)</desc>
  </programme>

2017_08_27_20_35_31

Maraba Televole should start 01 AM but my guide.xml says 21:45 but KODI shows correct time wrong date (28th) KODI PVR time offset= 0

  <programme start="20170827220000 +0000" stop="20170827224500 +0000" channel="Uçankuş TV">
    <title lang="tr">Maraba Televole</title>
  </programme>

So, If I add +03:00, kodi will show wrong, Date info is wrong for some channels on main DSMART TV GUIDE page. For example Nat Geo Channel's EPG info date is OK but Uçankuş wrong it says 24 Agust but no it is today. So your ini modification should just pull time correct date is your query date. Can you modify like that?

2017_08_27_20_45_47

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

Forget about my RaspberryPi questions. I started using linux mono on my remote RaspberryPi. What I did:

I have one Hobby Garden about 30km far away. There is an internet connection router and Raspberry there. Remotely I connected to my Raspberry and follow the instructions on this web site. I installed as HARDWARE FLOAT. I changed DSMART.ini (Just URLs) (it is working without php in linux now). I use 4 ini files for my EPG data. I transfered all of them to RaspberryPi. I installed wput package. I wrote a script like:

#!/bin/bash

/home/pi/.wg++/run.sh
cd /home/pi/.wg++/
wput guide.xml ftp://user:password@MY_FTP_SERVER_ADDRESS/WebGrab++/

I added above script to crontab for run every 6 hours. Works perfectly. But DSMART.INI still needs your final touch.

mkaand
Offline
mkaand's picture
Donator
Joined: 7 years
Last seen: 1 year

I think my mono installation is not OK or DMSART SSL is different. I tried to test tsltest.exe here is the results:

pi@raspberrypi:~ $ mono tlstest.exe https://www.nuget.org

https://www.nuget.org
pi@raspberrypi:~ $ mono tlstest.exe https://www.dsmart.com.tr

https://www.dsmart.com.tr
FAILED: #-2146232800
System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: Error while sending TLS Alert (Fatal:InternalError): System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer
  at System.Net.Sockets.Socket.EndReceive (System.IAsyncResult asyncResult) [0x00012] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
  at System.Net.Sockets.NetworkStream.EndRead (System.IAsyncResult asyncResult) [0x00057] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord (System.IAsyncResult asyncResult) [0x00037] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord (System.IAsyncResult ar, System.Boolean ignoreEmpty) [0x00000] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker (System.IAsyncResult result) [0x00071] in <d74853bc825444959b20ae5c72884af0>:0 
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslClientStream.EndNegotiateHandshake (System.IAsyncResult result) [0x00032] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (System.IAsyncResult asyncResult) [0x0000c] in <d74853bc825444959b20ae5c72884af0>:0  ---> System.IO.IOException: Unable to write data to the transport connection: The socket is not connected. ---> System.Net.Sockets.SocketException: The socket is not connected
  at System.Net.Sockets.Socket.BeginSend (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Net.Sockets.SocketFlags socketFlags, System.AsyncCallback callback, System.Object state) [0x00022] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
  at System.Net.Sockets.NetworkStream.BeginWrite (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.AsyncCallback callback, System.Object state) [0x0009b] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
   --- End of inner exception stack trace ---
  at System.Net.Sockets.NetworkStream.BeginWrite (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.AsyncCallback callback, System.Object state) [0x000e6] in <30b06929a39a4cd2a4ccaa70b756ffa0>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.BeginSendRecord (Mono.Security.Protocol.Tls.ContentType contentType, System.Byte[] recordData, System.AsyncCallback callback, System.Object state) [0x00023] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.SendRecord (Mono.Security.Protocol.Tls.ContentType contentType, System.Byte[] recordData) [0x00000] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.SendAlert (Mono.Security.Protocol.Tls.Alert alert) [0x00021] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.SendAlert (System.Exception& ex) [0x0001b] in <d74853bc825444959b20ae5c72884af0>:0 
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslStreamBase.EndNegotiateHandshake (Mono.Security.Protocol.Tls.SslStreamBase+InternalAsyncResult asyncResult) [0x00022] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslStreamBase.NegotiateHandshake () [0x0002b] in <d74853bc825444959b20ae5c72884af0>:0 
  at Mono.Security.Protocol.Tls.SslStreamBase.Write (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00064] in <d74853bc825444959b20ae5c72884af0>:0 
  at System.IO.StreamWriter.Flush (System.Boolean flushStream, System.Boolean flushEncoder) [0x0007e] in <c109e83bf6b6427c945b56620bfa750d>:0 
  at System.IO.StreamWriter.Flush () [0x00006] in <c109e83bf6b6427c945b56620bfa750d>:0 
  at TlsTest.GetStreamPage (System.String url) [0x000c2] in <275c587a7d944cc49858fd9ac0265885>:0 
  at TlsTest.Main (System.String[] args) [0x002a6] in <275c587a7d944cc49858fd9ac0265885>:0 
pi@raspberrypi:~ $ 

emrekaramannn
Offline
Joined: 7 years
Last seen: 7 years
mkaand wrote:

Hi All,

1 days ago I joined this forum and I tried to fix ini files for Turkish channels. Thanks to Blackbear199, I fixed all of them. I create special config file for best EPG coverage for all channels. I mean some of channels are in 2 or 3 ini files. I picked best of one. You should change channel name with your m3u file. I made an excel file to show which ini has the best EPG data for channels. Just DSMART uses php file. (it is in zip file)

I hope it will be useful for everyone. I found another EPG sources for tivibu channels. Maybe someone can create another ini for tivibu:

http://www.tivibuspor.com.tr/yayin-akisi

Another one for beIN Sports Turkey : http://tr.beinsports.com/yayin-akisi/beinsports

 

 

 

Hi all,

It seems quite complicated for a casual user. How can i upload to iptv program? PHP file does not work on linux server. No epg appears, please help.

konrul
Offline
Joined: 3 years
Last seen: 3 years
mkaand wrote:

Hi All,
1 days ago I joined this forum and I tried to fix ini files for Turkish channels. Thanks to Blackbear199, I fixed all of them. I create special config file for best EPG coverage for all channels. I mean some of channels are in 2 or 3 ini files. I picked best of one. You should change channel name with your m3u file. I made an excel file to show which ini has the best EPG data for channels. Just DSMART uses php file. (it is in zip file)
I hope it will be useful for everyone. I found another EPG sources for tivibu channels. Maybe someone can create another ini for tivibu:
http://www.tivibuspor.com.tr/yayin-akisi
Another one for beIN Sports Turkey : http://tr.beinsports.com/yayin-akisi/beinsports

Hello,

Thank you for your post. I was almost losing my hope to solve the EPG problem for my IPTV. Thanks to you, this post made me hopeful of solving my Turkish EPG resource problem.

Could you please help me with how to use these files and programs?

I really appreciate any help you can provide.

mat8861
Offline
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 day

can you post your webgrab_log.txt so we can see what's wrong ? thanks

konrul
Offline
Joined: 3 years
Last seen: 3 years

Hello mat8861, did you ask me?

If so, I am a real newbie here on this program and related to EPG-IPTV stuff... So I need basics and how to use it...

mat8861
Offline
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 day

well you can install wg++, run it and see the results.It's pretty easy, if you need help just follow documentation or ask here.

konrul
Offline
Joined: 3 years
Last seen: 3 years
mat8861 wrote:

can you post your webgrab_log.txt so we can see what's wrong ? thanks

Hello, here is the log file

I am totally new, let me clear my need:

I want to get EPG data with this program. Then I want to match channels and EPG data. I don't know how to get XML file with this program. I couldn't find any tutorial that explain easily. If you can explain "how to" with text or video /any kind of way i will be happy.

Thanks in advance.

The file has been updated, previous one was wrong file.

Attachments: 
mat8861
Offline
WG++ Team memberDonator
Joined: 9 years
Last seen: 1 day

i don't know if those siteini posted are working, any way set you config as example here:
https://github.com/SilentButeo2/webgrabplus-siteinipack/blob/master/site...
then add channels from https://github.com/SilentButeo2/webgrabplus-siteinipack/tree/master/site...
and do a run. as simple suer you cannot run encrypted siteini (see wglicense log.)

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl